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ABSTRACT 

This  thesis  proposes  a  methodology  for  measuring  the  extent  to 
which  basic  systems  engineering  principles  and  practices  are 
followed  during  a  system  development  effort,  as  well  as  for 
measuring  other  characteristics  of  the  system  development 
process,  in  attempts  to  relate  the  success  of  the  development 
effort  to  the  degree  of  systems  engineering.  The  approach  enables 
the  evaluation  and  rating  of  case  studies,  regardless  of  system 
type  or  size,  based  on  figures  of  merit  and  rating  criteria 
developed  for  each  of  five  characteristics  of  the  system 
development  process  as  modeled.  The  scoring  methodology  is 
demonstrated  using  six  case  studies  of  recent  commercial  and 
military  aircraft  development  programs,  including  the  Boeing  777, 
Lockheed  F-117,  Northrop  B-2,  McDonnell  Douglas  C-17,  Lear jet 
Model  60,  and  McDonnell  Douglas  MD-11.  The  primary  purpose  for 
conducting  this  effort  is  to  provide  a  tool  to  enhance  systems 
engineering  understanding  and  education. 
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INTRODUCTION 


The  attempt  to  understand  the  effectiveness  of  performing  systems 
engineering  practices  during  the  development  of  complex  systems 
is  hindered  by  the  lack  of  a  common  definition  of  systems 
engineering  and  a  methodology  to  measure  its  degree  of 
application.  What  has  also  been  lacking  is  a  top-level  model  that 
relates  various  characteristics  of  the  system  design  process  to 
the  results  of  the  development,  namely,  technical  performance, 
cost  performance,  and  schedule  performance.  These  outcomes, 
comprising  the  performance  measure,  define  the  extent  of  overall 
success  of  the  system  development  effort. 

This  thesis  proposes  a  methodology  for  rating  case  studies  of 
past  system  development  efforts  based  on  criteria  developed  for 
each  characteristic  of  the  system  development  process. 
Furthermore,  it  is  proposed  that  the  scores  may  be  used  to 
perform  a  comparison  between  different  case  studies  in  order  to 
explore  the  empirical  relationships  between  the  different 
characterstics  of  the  system  development  process.  These 
characteristics,  in  addition  to  the  performance,  include  systems 
engineering  fundamentals ,  development  environment ,  design 
difficulty ,  and  resources  to  accomplish  system  development. 

This  effort  is  intended  primarily  to  provide  a  tool  to  enhance 
systems  engineering  understanding  and  education. 
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CHAPTER  1 

EVALUATING  SYSTEMS  ENGINEERING 

" Systems  engineering"  is  a  relatively  common  term  used  in  the 
engineering  world.  Despite  the  recognition,  there  is  no  exact, 
universal  definition  of  what  systems  engineering  is  and  what  it 
denotes.  However,  there  appears  to  be  a  common  belief  that 
systems  engineering,  whatever  its  specific  definition,  is  an 
important  if  not  crucial  element  in  the  successful  development  of 
complex  entities  composed  of  hardware,  software,  and  people.  Over 
the  past  20  years,  there  has  been  an  emerging  consensus  as  to  the 
definition  of  systems  engineering  and  the  importance  of  following 
its  principles  as  part  of  the  system  design  process.  However, 
what  is  the  basis  of  the  belief  in  the  importance  of  systems 
engineering?  Most  case  studies  describing  a  system  development 
never  mention  the  word  systems  engineering  or  outline  the 
application  of  whatever  it  constitutes.  To  be  able  to  determine 
whether  or  not  systems  engineering  is  actually  an  important 
factor  in  the  development  of  systems,  there  should  be  a  standard, 
concise  way  to  measure  and  represent  the  degree  to  which  the 
discipline  is  followed  during  development.  Since  systems 
engineering  involves  a  wide  range  of  concerns,  and  it  means 
different  things  to  different  people,  evaluating  it  in  a  manner 
that  everyone  can  agree  with  becomes  a  tremendous  challenge.  The 
first  step  towards  being  able  to  measure  and  evaluate  impact  is 
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to  understand  what  systems  engineering  fundamentally  is. 

Definitions  now  abound.  Five  of  them  are  presented  below: 

According  to  Dr.  A.  Wayne  Wymore  in  Model  Based  Systems 
Engineering,  "Systems  engineering  is  the  intellectual, 
academic,  and  professional  discipline  the  principal  concern  of 
which  is  the  responsibility  to  ensure  that  all  reguirements 
for  a  system  are  satisfied  throughout  the  life  cycle  of  the 
system. " 

According  to  the  draft  version  of  MIL-STD-499B,  "Systems 
engineering  is  the  management  and  technical  process  that 
controls  all  engineering  activities  throughout  the  life  cycle 
in  order  to  achieve  an  optimum  balance  of  all  system  elements 
to  ensure  satisfaction  of  system  requirements  while  providing 
the  highest  degree  of  mission  success.  It  has  two  main 
activities:  (a)  interpreting  the  customer's  needs  and 
translating  them  into  a  set  of  requirements  that  can  be  met  by 
individual  design  and  specialty  disciplines  and  (b)  validating 
that  the  system  satisfies  the  customer's  needs  through 
analysis,  simulation,  and  testing." 

According  to  the  Defense  Systems  Management  College  Systems 
Engineering  Management  Guide,  "Systems  engineering  is  the 
management  function  which  controls  the  total  system 
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development  effort  for  the  purpose  of  achieving  an  optimum 
balance  of  system  elements.  It  is  a  process  which  transforms 
an  operational  need  into  a  description  of  system  parameters 
and  integrates  those  parameters  to  optimize  the  overall  system 
effectiveness. " 

According  to  Successful  Systems  Engineering  For  Engineers  and 
Managers  by  Norman  B.  Reilly,  "Systems  engineering  is  the 
systematic  application  of  proven  standards,  procedures,  and 
tools  to  the  technical  organization,  control,  and 
establishment  of  system  requirements,  system  design,  system 
management,  system  fabrication,  system  integration,  system 
testing,  and  system  logistics  support." 

According  to  the  draft  IEEE  P1220  Standard  for  Systems 
Engineering ,  systems  engineering  is  an  interdisciplinary 
collaborative  approach  to  derive,  evolve,  and  verify  a  life- 
cycle  balanced  solution  that  meets  customer  and  public 
acceptability. 

The  definitions  say  similar  things.  For  the  most  part, 
differences  in  definition  reside  in  the  terminology  and  details 
of  what  specific  activities  it  constitutes,  the  scope  to  which  it 
is  applied,  and  when  in  the  life  cycle  it  is  appropriate.  Since 
the  definitions  above  are  broad,  they  do  not  provide  a  concrete 
understanding  to  lay  persons  of  what  systems  engineering 
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involves.  Nor  do  they  necessarily  point  to  a  set  of  criteria  by 
which  a  system  development  program  could  be  evaluated  as  to  the 
degree  to  which  systems  engineering  is  followed.  Despite  the 
differences  in  detail  and  emphasis,  philosophically  they  are  the 
same.  The  same  can  be  said  about  the  different  outlines  of  the 
system  life  cycle.  Five  versions  from  different  authors  and 
organizations  are  provided  in  Figure  1.1. 

To  be  more  specific  and  concrete,  a  list  could  be  generated  of 
the  numerous  tasks,  tools,  and  technigues,  both  technical  and 
managerial,  that  are  considered  "systems  engineering"  by 
different  people  and  organizations,  and  such  a  list  could  enhance 
common  understanding.  Unfortunately,  no  one  list  would  be 
agreeable  to  all  people,  even  systems  engineers.  Furthermore, 
while  there  are  many  different  tasks  performed  under  the  label  of 
"systems  engineering,"  some  are  not  appropriate  for  application 
under  all  circumstances.  Degree  of  task  application,  the 
formality  in  which  the  element  is  carried  out,  and  how  well  the 
activity  is  performed  are  considerations  affecting  systems 
engineering  effectiveness  in  a  particular  development  process. 
Furthermore,  the  environment  in  which  development  occurs  greatly 
impacts  and  can  even  determine  the  success  or  failure  of  system 
development  by  affecting  the  process  itself.  Additionally,  the 
challenge  and  magnitude  of  the  system  design  is  another  factor 
impacting  system  performance.  Therefore,  it  is  not  possible  to 
merely  present  a  complete  "laundry"  list  of  systems  engineering 


Figure  1.1  System  life  cycle  phases  -  different  versions 
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tasks,  tools,  and  techniques  and  simply  claim  that  if  a  systems 
development  effort  performs  them  all,  success  will  result.  Since 
design  is  an  iterative  process  by  which  the  elements  of  systems 
engineering  are  applied  most  effectively  in  a  logical, 
disciplined  manner  throughout  the  life  cycle  of  the  system, 
development  must  be  viewed  and  evaluated  in  several  ways  to 
obtain  a  clearer  picture  of  the  relationship  between  the  results 
of  system  development  and  the  characteristics  of  the  process 
employed  to  get  there. 

Application  of  systems  engineering  principles  and  activities  must 
be  tailored  to  each  particular  system  development  effort. 

However,  there  are  no  known  immutable,  universal  templates  or 
algorithms  for  applying  each  and  every  systems  engineering 
element.  Much  like  designing  a  complex  system  itself,  deciding  on 
the  implementation  of  systems  engineering  tasks,  tools,  and 
techniques  involves  many  decisions  requiring  common  sense  and 
educated  evaluation  of  numerous  tradeoffs  in  order  to  achieve  the 
best  outcomes  possible.  Despite  the  qualitative  judgments 
involved  with  applying  and  carrying  out  systems  engineering,  it 
is  proposed  that  a  credible  methodology  can  be  utilized  to 
measure  or  rate  its  magnitude  of  application  on  development 
efforts  already  completed.  The  methodology  can  also  be  used  to 
rate  the  magnitude  of  involvement  of  the  other  characteristics  of 
system  development,  such  as  environment,  design  difficulty,  and 
required  resources.  Using  these  results  alone  can  be  useful  in 
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illustrating  what  systems  engineering  involves  for  the  benefit  of 
engineering  students.  However,  these  ratings  might  also  be  useful 
in  empirically  exploring  the  relationships  among  the  system 
development  process  characteristics  and  the  end  results  of  an 
effort. 

How,  then,  can  a  system  development  effort  be  evaluated  as  to  the 
extent  of  systems  engineering  practices  followed  and  their 
effectiveness  as  part  of  the  design  process?  This  thesis  proposes 
the  following  approach.  First,  an  evaluation  methodology  is 
developed  through  the  creation  of  figures  of  merit  and  a  rating 
scale  for  each  of  the  five  characteristics  of  a  system 
development  effort.  These  characteristics  consist  of  (1) 
performance  involving  technical,  cost,  and  schedule  performance; 

( 2 )  systems  engineering  fundamentals  describing  eleven  crucial 
principles  and  practices  that  should  be  considered  for 
application  in  any  development;  (3)  eleven  aspects  of  the 
development  environment  that  are  suggested  to  be  conducive  to 
supporting  successful  system  development;  (4)  design  difficulty 
consisting  of  six  figures  of  merit  addressing  various  aspects  of 
technical  complexity,  and  (5)  resources  in  terms  of  cost,  time 
and  infrastructure  to  complete  development.  Within  each 
characteristic,  the  figures  of  merit  or  elements  are  selected  to 
be  distinct  and  measurable  and  have  a  minimum  amount  of 
dependency  between  each  other.  Scoring  criteria  associated  with 
each  of  the  figures  of  merit  and  elements  are  generic,  and 
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therefore  they  are  appropriate  for  any  type  and  size  of  system. 
This  allows  "apples-to-apples"  comparisons  of  the  quality  of  the 
development  processes  for  different  systems. 

Second,  these  five  characteristics  are  related  to  each  other  by 
the  model  in  Figure  1.2.  This  basic  representation  shows  that 
systems  engineering  fundamentals,  development  environment ,  design 
difficulty ,  and  resources  comprise  a  system  development  effort, 
and  they  interact  to  produce  the  performance  outcomes.  This 
simple  model  represents  numerous  complex  interactions,  but  it 
does  not  describe  how  they  occur.  It  is  not  known  whether  all  or 
some  of  the  characteristics  are  additive  or  multiplicative,  or 
whether  they  can  be  combined  at  all.  The  major  challenge  is  to 
attempt  to  understand  how  they  react  at  a  macro-level,  if  that  is 
even  possible.  The  hypothesis  being  pursued  is  that  the 
characteristics  can  be  rated  with  meaningful  single  numerical 
scores,  thereby  providing  a  shortcut  means  of  analyzing  the 
complex  processes  involved.  Although  the  system  development  model 
is  not  a  purely  theoretically  derived  construct,  there  are  sound 
bases  for  much  of  the  descriptive  model.  For  example,  the  systems 
engineering  fundamentals  figures  of  merit  correspond  to  the 
managerial  functions  of  planning,  organizing,  leading, 
controlling,  and  coordinating  as  defined  in  the  Traditional 
Management  School  model.  Furthermore,  the  development  environment 
figures  of  merit  correspond  closely  with  many  aspects  of 
organizational  theory. 
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Third,  in  order  to  demonstrate  the  ratings  methodology  and 
provide  data  for  analysis,  six  case  studies  are  presented  and 
evaluated  in  accordance  with  the  figures  of  merit  rating 
criteria.  The  subject  of  the  six  cases  is  aircraft  developed  in 
the  United  States  within  the  past  15  years.  This  topic  was  chosen 
because  ( 1 )  aircraft  are  some  of  the  most  complex  systems 
developed  today,  (2)  a  reasonably  large  number  of  companies 
develop  aircraft  and  each  does  it  differently,  and  (3)  aircraft 
development  is  a  somewhat  mature  process. 


Figure  1.2  System  Development  Model 


Design 

Difficulty 
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The  fourth  part  of  the  approach  is  the  compilation  and  analysis 
of  the  results.  The  individual  scores  of  the  figures  of  merit  and 
elements  are  summed  together  to  produce  one  numerical  score  for 
each  of  the  five  characteristics.  This  approach  employs  a  simple 
multi-attribute  rating  technique.  The  resulting  numbers  are  used 
in  an  attempt  to  uncover  relationships  among  the  characteristics, 
with  particular  emphasis  on  identifying  a  correlation  between 
systems  engineering  fundamentals  and  the  performance  scores.  The 
meaning  of  a  relatively  higher  score  for  performance  is  not  that 
one  system  is  necessarily  better  than  the  other.  Rather,  it  is  a 
quality  or  success  index  of  the  development  process  itself . 

The  selection  of  the  characteristics  and  their  figures  of  merit 
were  developed  from  a  variety  of  sources.  Performance 
incorporates  the  traditional  measures  (technical,  cost,  and 
schedule  performance)  of  assessing  the  success  of  a  development 
program.  The  systems  engineering  fundamentals  represent  the 
summarization  and  synthesis  of  a  detailed  list  of  principles, 
tasks,  and  techniques  compiled  from  various  systems  engineering 
books  and  papers.  (See  references.)  The  development  environment 
figures  of  merit  were  devised  based  on  the  author's  experience  as 
well  as  organizational  studies  (Quinn  &  Rohrbaugh,  1983;  Elmes  & 
Wilemon,  1992).  They  were  also  chosen  based  on  their  abilities  to 
be  measured  to  a  reasonable  degree.  The  elements  and  rating 
methodology  for  both  design  difficulty  and  resources  were  taken, 
with  some  modifications,  from  William  L.  Chapman's  1994  Ph.D. 
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dissertation  at  the  University  of  Arizona,  "The  System  Design 
Process  is  Intractable,  But  Robust."  As  part  of  his  dissertation, 
he  rated  18  case  studies  covering  a  wide  range  of  efforts 
throughout  history  according  to  the  two  characteristics.  He  then 
plotted  the  cases  on  a  two-axis  graph  to  show  how  they  compared 
to  one  another. 

The  approach  and  model  design  just  presented  can  be  viewed  as 
logical,  and  they  represent  a  first  step  towards  systematically 
investigating  the  relationships  among  the  high-level  system 
development  characteristics  and  the  results  of  a  development 
effort.  However,  there  are  a  variety  of  questions  and  issues 
concerning  the  validity  of  the  approach  as  related  to  the 
qualitative  nature  of  many  of  the  criteria,  design  of  the 
numerical  rating  scale,  independence  of  the  figures  of  merit, 
subjectivity  of  the  process,  bias  in  the  case  studies,  and 
mathematically  combining  the  figures  of  merit  scores. 

First  of  all,  the  assignment  of  scores  for  the  figures  of  merit 
and  elements  is  based  on  rating  criteria  for  all  five  system 
development  characteristics,  presented  in  Chapter  2.  The  topics 
addressed  by  the  figures  of  merit  for  systems  engineering 
fundamentals  and  development  environment  do  not  lend  themselves 
to  the  formation  of  quantitative  criteria  for  the  most  part. 
Instead,  these  criteria  are  mostly  qualitative,  and  many  of  them 
are  imprecise.  Given  the  complexity  of  the  topics  being 
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evaluated,  the  criteria  must  be  at  a  very  high  level  or  else  be 
made  extremely  detailed.  Making  them  detailed  reduces  their 
usefulness  for  educational  purposes. 

The  zero  to  ten  point  scale  used  in  a  majority  of  the  figures  of 
merit  rating  criteria  was  selected  because  it  allows  easy 
separation  of  ratings  into  three  ranges  of  high,  medium,  and  low. 
Given  the  gualitative  nature  of  most  of  the  criteria,  subjective 
judgments  can  be  easily  assigned  among  three  categories  based  on 
a  reasonable  amount  of  information  provided  to  the  evaluator . 
Furthermore,  allowing  the  choice  of  several  points  within  each  of 
the  three  categories  provides  the  rater  the  ability  to  score  with 
greater  fidelity  if  the  degree  of  information  detail  is  provided. 

While  the  case  studies  were  researched  rather  extensively,  the 
information  gathered  was  obtained  from  limited  sources. 

Therefore,  there  may  be  unintended  biases,  omissions,  or 
inaccuracies  which  may  impact  the  ratings  for  particular  figures 
of  merit. 

An  issue  of  fundamental  importance  is  whether  or  not  the  proposed 
figures  of  merit  ratings  slimmed  into  single  higher  level 
characteristic  scores  have  validity  and  significance  as  generic 
measures,  thereby  allowing  meaningful  comparison  between  systems. 
This  issue  depends  on  the  degree  of  dependence  between  the 
figures  of  merit  within  a  characteristic,  the  extent  to  which  the 
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figures  of  merit  and  elements  completely  define  the  factors  at 
work  within  a  characteristic,  and  the  appropriateness  of  the 
relative  weighting  within  each  characteristic.  While  some 
dependencies  do  exist  between  the  figures  of  merit,  most  can  be 
considered  as  minimal  based  on  a  simple  cause-effect  analysis. 
Also,  the  characteristics  can  be  reasonably  regarded  as  not 
leaving  out  major  high-level  factors  since  the  figures  of  merit 
address  all  the  major  areas  identified  in  appropriate  literature. 

The  scoring  method  for  the  first  three  characteristics  assumes 
that  most  figures  of  merit  have  egual  weight  within,  since  the 
relative  importance  of  each  is  not  understood  in  a  quantitative 
way.  The  hope  is  that  insights  into  relative  weighting  may  be 
gained  by  applying  the  approach  outlined  in  this  thesis  to  a 
sufficiently  large  number  of  cases.  The  exceptions  to  the  equal 
weighting  assumption  are  the  first  two  figures  of  merit  in  the 
systems  engineering  fundamentals  characteristic .  They  are  each 
weighted  twice  their  ratings  based  on  studies  that  have  shown 
that  the  majority  of  the  life  cycle  cost  of  a  system  is 
determined  in  the  requirements  and  concept  development  phases. 
[Zangwill ,  1992]  Of  all  the  figures  of  merit  in  the  systems 
engineering  fundamentals  characteristic,  these  two  are  most 
associated  with  the  early  phases  of  a  program.  While  this  overall 
weighting  structure  does  warrant  further  investigation  and 
refinement,  it  is  meant  to  be  a  starting  point. 
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An  important  point  to  make  is  that  this  thesis  is  not  attempting 
to  explore  in  detail  the  combinatorial  issues  surrounding  the 
summation  of  the  ratings.  While  there  is  a  huge  amount  of 
literature  dealing  with  mathematical  and  behavioral  aggregation 
and  multi-criteria  decision  making,  delving  into  those  issues 
would  take  this  thesis  beyond  what  is  needed  at  this  point  in  the 
development  of  the  approach.  Instead,  as  mentioned  previously, 
the  figures  of  merit  are  believed  to  be  reasonably  independent, 
thereby  supporting  the  basic  validity  of  the  approach. 
Furthermore,  only  simple  graphical  analysis  technigues  are  used 
to  analyze  the  data  generated.  Due  to  the  low  number  of  data 
points  generated,  sophisticated  statistical  methods  are  of 
minimal  usefulness. 

While  the  focus  of  this  thesis  is  systems  engineering  and  its 
impact  in  the  system  development  process,  there  is  some  relation 
to  the  area  of  organizational  effectiveness.  Although  the  model 
and  analysis  method  presented  are  not  oriented  towards  any 
specific  organizational  structure,  they  take  into  account 
organizational  factors  in  the  systems  engineering  fundamentals 
and  development  environment  characteristics.  In  fact,  some  of  the 
development  environment  figures  of  merit  correspond  almost 
directly  with  organizational  effectiveness  measures. 

Many  studies  have  been  conducted  attempting  to  determine, 
measure,  and  evaluate  the  characteristics  of  successful 
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organizations,  and  many  case  studies  have  been  developed 
describing  successful  and  unsuccessful  products  and  how  they  were 
developed.  Furthermore,  studies  of  the  effectiveness  of 
concurrent  engineering,  a  development  approach  which  can  be 
viewed  as  good  systems  engineering,  identify  most  of  the  same 
types  of  issues  addressed  in  the  systems  engineering 
fundamentals .  (See  Zangwill,  1992.)  However,  to  the  author's 
knowledge,  none  have  structured  the  development  process  in  the 
same  way  as  presented  in  this  thesis  or  attempted  to  uncover  high 
level  relationships  between  characteristics  of  system 
development.  Furthermore,  no  other  studies  have  identified 
systems  engineering  fundamentals  as  recognizable  elements  that 
can  be  evaluated  as  a  group  and  represented  with  a  single 
numerical  rating.  This  thesis  proposes  such  a  methodology. 

The  outline  of  the  evaluation  methodology  showing  the  system 
development  characteristics  and  lists  of  the  figures  of  merit  are 
presented  in  the  following  sections. 

Performance 

Performance  is  composed  of  measures  used  to  assess  the  degree  of 
success  of  the  overall  development  process  of  a  system.  The 
figures  of  merit  consist  of  the  following: 

A.  Technical  performance  at  initial  deliveries. 

B.  Technical  performance  one  to  two  years  after  delivery. 
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C.  Cost  performance  during  full-scale  engineering  development 
( FSED ) . 

D.  Schedule  performance  during  FSED. 

Systems  Engineering  Fundamentals 

The  approach  to  measure  the  degree  of  system  engineering  followed 
during  development  involves  identifying  a  top-level  list  of  key 
principles,  activities  and  tasks  that  should  be  considered  during 
any  development  program  of  reasonable  complexity.  This  approach 
is  in  contrast  to  trying  to  develop  a  detailed,  all  encompassing 
list.  As  part  of  the  rating  process  for  each  case  study,  the 
fundamental  systems  engineering  practices  are  evaluated  as  to 
whether  or  not  they  were  accomplished,  to  what  extent  they  were 
completed,  and  how  well  these  fundamental  elements  were  carried 
out.  This  involves  not  just  the  issue  of  completeness  and  the 
expertise  displayed,  but  also  the  appropriateness  of  application 
and  the  proper  timing  in  which  system  engineering  elements  are 
applied  and  carried  out. 

The  following  is  a  list  of  fundamental  practices  or  elements  of 
good  systems  engineering.  In  the  author's  opinion,  all  are 
accomplished  as  part  of  effective  systems  engineering.  However, 
the  extent  and  formality  of  the  application  depends  on  the 
particulars  of  the  system  being  developed.  The  tasks  associated 
with  these  figures  of  merit  are  not  necessarily  performed  by 
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officially  designated  systems  engineers: 

A.  Requirements  development. 

B.  Incipient  system  design. 

C.  Evaluating  alternative  concepts  and  designs. 

D.  Make-or-buy  decision. 

E.  Validation. 

F.  Verification  and  integrated  testing. 

G.  Configuration  management. 

H.  Manufacturing  considerations. 

I .  Systems  integration  and  technical  management . 

J.  Life  cycle  considerations. 

K.  Program  management. 

Development  Environment 

The  third  aspect  of  evaluating  the  systems  development  process  is 
the  environment  in  which  it  takes  place.  The  environment  is 
probably  an  equally  crucial  factor  for  success.  Even  expertly 
performed  systems  engineering  practices  can  be  overcome  by 
negative  environmental  factors.  However,  a  positive  environment 
cannot  make  up  for  inadequate  or  poorly  performed  systems 
engineering.  Both  systems  engineering  fundamentals  and 
development  environment  work  together  and  impact  each  other  in 
complex  ways. 
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Most  of  the  elements  of  this  third  aspect  are  beyond  the  direct 
control  of  the  systems  engineer,  chief  engineer,  and  the  program 
manager.  Some  are  under  the  control  of  those  in  authority  above 
the  effort  or  totally  external  to  it.  However,  if  these  elements 
are  favorable,  they  can  enhance  the  effectiveness  of  systems 
engineering  practices  and  contribute  to  development  success.  They 
are  as  follows: 

A.  Emphasis  on  the  customer. 

B.  Stability  of  requirements  and  configuration. 

C.  Funding  and  workforce-level  stability. 

D.  Strong  support  of  program. 

E.  Continuity  of  core  development  team. 

F.  Stability  of  organizational  structure. 

G.  Cooperation  among  all  stakeholders. 

H.  Effective  communication  within  team. 

I.  Flexibility  and  autonomy. 

J.  Workforce  expertise  and  experience. 

K.  Accountability  for  system  performance. 

Design  Difficulty 

Design  difficulty  essentially  represents  different  aspects  of 
technical  complexity  and  scope  of  effort.  The  following  is  a  list 
of  the  elements: 

A.  Design  type. 

B.  Knowledge  complexity. 
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C.  Steps  (needed  to  complete  the  design). 

D.  Quality  implementation  effort. 

E.  Extent  of  manufacturing  operations. 

F.  Selling  price  constraint. 

Resources 

The  resources  characteristic  refers  to  those  items  needed  to 
complete  the  design,  test,  and  fabrication  of  the  system.  The 
following  is  a  list  of  the  resources  elements: 

A.  Cost. 

B .  Time . 

C.  Infrastructure. 

In  the  next  chapter,  a  description  for  each  figure  of  merit  and 
element  and  the  methodology  for  evaluating  and  rating  a  system 
development  effort  are  presented.  The  methodology  involves 
criteria  and  scoring  rules  developed  for  each  figure  of  merit  and 
element,  and  it  results  in  a  single  number  score  for  each  of  the 
five  characteristics. 
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CHAPTER  2 

EVALUATION  AND  SCORING  METHODOLOGY 


This  chapter  contains  the  methodology  for  evaluating  system 
design  case  studies  and  assigning  guantitative  scores  to  them. 

The  five  development  effort  characteristics  are  presented  with 
the  figures  of  merit  or  elements  which  define  them  as  well  as  the 
criteria  for  assigning  scores.  The  figures  of  merit  scores  for 
each  case  are  added  together,  resulting  in  a  single  numerical 
score  for  each  of  the  characteristics.  The  first  three 
characteristics  presented,  performance ,  systems  engineering 
fundamentals ,  and  development  environment,  are  positive  measures, 
in  that  the  higher  the  score,  the  better.  The  design  difficulty 
and  resources  characteristics,  however,  merely  describe  the 
system  design  and  do  not  reflect  relative  "goodness"  of  the 
design  effort  or  the  design. 

Performance 

A.  Technical  performance  -  initial.  This  is  defined  as  compliance 
with  customer  performance  reguirements  and  specifications  and 
overall  customer  satisfaction  at  time  of  initial  system 
deliveries. 

•  7-10  points  are  given  for  a  highly  successful  system,  in 
that  the  system  achieves  all  or  nearly  all  key  technical 
performance  requirements  at  time  of  initial  system  delivery . 
There  is  a  high  degree  of  customer  satisfaction. 
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•  4-6  points  are  given  for  a  moderately  successful  system,  in 
that  the  system  meets  a  majority  of  key  technical 
performance  requirements,  rendering  it  useful  to  the 
customer.  However,  operational  performance  of  some 
subsystems  is  less  than  expected.  The  result  is  a  moderate 
degree  of  customer  satisfaction. 

•  0-3  points  are  given  for  an  unsuccessful  system,  in  that  the 
system  fails  to  meet  significant  technical  performance 
requirements  at  time  of  initial  system  delivery,  rendering 
it  unusable  by  the  customer  as  originally  intended.  The 
result  is  low  customer  satisfaction. 


B.  Technical  performance  -  mature.  This  is  defined  as  compliance 
with  customer  performance  requirements  and  specifications  and 
overall  customer  satisfaction,  one  to  two  years  after  system 
delivery. 

•  7-10  points  are  given  for  a  highly  successful  system,  in 
that  the  system  achieves  all  or  nearly  all  key  technical 
performance  requirements.  There  is  a  high  degree  of 
customer  satisfaction. 

•  4-6  points  are  given  for  a  moderately  successful  system,  in 
that  the  system  meets  a  majority  of  key  technical 
performance  requirements  rendering  it  useful  to  the 
customer.  The  result  is  a  moderate  degree  of  customer 
satisfaction. 

•  0-3  points  are  given  for  an  unsuccessful  system,  in  that  the 
system  fails  to  meet  significant  technical  performance 
requirements,  rendering  it  unusable  by  the  customer  as 
originally  intended. 


C.  Cost  performance.  This  deals  with  how  close  the  full-scale 
engineering  development  (FSED)  effort  meets  the  budget; 
production  costs  are  not  considered.  This  is  measured  using 
percentage  cost  growth  (overrun)  of  the  original  FSED  baseline 
estimate.  This  measure  is  from  the  point  of  view  of  the 
funding  source. 
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•  9-10  points  are  given  for  an  effort  conducted  below  the 
baselined  development  cost  requirement  to  no  more  than  5 
percent  cost  growth  (overrun). 

•  7-8  points  are  given  for  an  effort  with  6  to  15  percent 
cost  growth  (overrun)  of  the  baselined  development  cost 
requirement. 

•  5-6  points  are  given  for  an  effort  with  16  to  35  percent 
cost  growth  (overrun) . 

•  3-4  points  are  given  for  an  effort  with  36  to  75  percent 
cost  growth  (overrun) . 

•  0-2  points  are  given  for  an  effort  with  greater  than  75 
percent  cost  growth  (overrun). 


D.  Schedule  performance.  This  is  defined  as  the  percentage  of 
time  that  the  FSED  effort  is  overdue,  with  the  effort  defined 
to  take  place  from  the  start  of  FSED  to  the  delivery  to  the 
customer  of  the  first  production  unit.  This  is  determined  by 
taking  the  length  of  time  from  the  start  of  FSED  to 
planned  customer  delivery  of  the  first  production  unit 
(planned  FSED  length),  taking  the  length  of  time  from  the 
start  of  FSED  to  the  actual  delivery  of  the  first  unit,  take 
the  difference  between  the  two  (asstiming  the  actual  is  longer 
than  the  planned  length),  and  dividing  the  difference  by  the 
planned  FSED  length. 

•  9-10  points  are  given  for  on-time  performance  or  not  more 
than  2  percent  overdue  (baselined  development  schedule). 

•  7-8  points  are  given  for  3  percent  or  more  overdue  but  less 
than  10  percent  overdue. 

•  5-6  points  are  given  for  10  percent  or  more  overdue  but  less 
than  20  percent  overdue. 

•  3-4  points  are  given  for  20  percent  overdue  or  more  but  less 
than  50  percent  overdue. 
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•  0-2  points  are  given  for  50  percent  or  more  overdue. 

The  ratings  and  scores  are  provided  in  a  table  at  the  end  of  each 
case  study  similar  to  that  illustrated  in  Table  2.1. 


Table  2.1  Performance  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

TECHNICAL  PERFORMANCE  - 
INITIAL 

0-10 

1 

TECHNICAL  PERFORMANCE  - 
MATURE 

0-10 

1 

COST  PERFORMANCE 

0-10 

1 

SCHEDULE  PERFORMANCE 

0-10 

1 

PERFORMANCE  TOTAL 

I 

0-40 

Systems  Engineering  Fundamentals 


A.  Requirements  development.  This  is  defined  as  understanding 
customer  needs,  properly  stating  the  problem,  and  accurately 
specifying  the  requirements  that  define  what  the  system  must 


do . 

•  7-10  points  are  given  for  accurate  and  thorough 
understanding  and  documentation  of  the  customer's  problems 
and  needs,  and  for  the  documentation  of  system-level 
requirements  in  a  single  specification. 

•  4-6  points  are  given  for  reasonably  accurate  and  thorough 
understanding  of  the  customer's  problems  and  needs  in  a 
requirements  document,  but  significant  adjustments  are 
required. 

•  0-3  points  are  given  for  inaccurate  and  incomplete 
definition  of  the  customer's  problems  and  needs. 
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B.  Incipient  system  design.  This  consists  of  defining  system 

concept  models  at  the  beginning  of  the  effort,  identifying  and 
organizing  the  hierarchy  of  functions  (functional 
decomposition)  to  be  performed  by  the  system  based  on  the  top- 
level  performance  objectives,  defining  the  system  constraints 
and  interfaces,  defining  the  hierarchy  of  physical  elements 
(physical  decomposition),  and  allocating  detailed  performance 
and  interface  requirements  to  the  physical  elements 
( subsystems ) . 

•  7-10  points  are  given  for  clearly  defined  top-level  system 
concept  models  that  are  decomposed  into  functions  or 
elements,  with  well  defined  interfaces  and  detailed 
requirements  allocations  to  the  decomposed  items. 
Furthermore,  these  are  completed  before  FSED. 

•  4-6  points  are  given  for  marginally  defined  top-level 
system  concept  models  that  are  decomposed  into  functions  or 
elements,  with  adequately  defined  interfaces  and 
requirements  allocations  to  the  decomposed  items. 
Furthermore,  these  are  completed  before  FSED. 

•  0-3  points  are  given  for  inadequate  top-level  modelling  of 
the  system  concept,  insufficient  or  inappropriate  interface 
definitions,  and  poor  requirements  allocation,  or  the 
failure  to  adequately  complete  such  activities  before  FSED. 


C.  Evaluating  alternative  concepts  and  designs.  This  refers  to 
evaluating  the  relative  merit  of  alternative  concepts  and 
designs  using  a  design  tradeoff  methodology  (formal  or 
informal)  that  uses  results  of  analyses,  simulation  model 
testing  and/or  physical  model  testing. 

•  7-10  points  are  given  for  thoughtfully  evaluating  and 
deciding  among  alternative  concepts  and/or  subsystem 
designs  using  a  formal  or  informal  design  tradeoff 
methodology  utilizing  results  of  analyses,  simulation  model 
testing  and/or  physical  model  testing. 
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•  4-6  points  are  given  for  limited  consideration  and  analysis 
of  alternative  concepts  and  subsystem  designs  using  a  formal 
or  informal  design  tradeoff  methodology  utilizing  results  of 
analyses,  simulation  model  testing  and/or  physical  model 
testing. 

•  0-3  points  are  given  for  little  or  no  consideration  and 
analysis  of  alternative  concepts  and  subsystem  designs. 


D.  Make-or-buy  decision.  This  is  the  act  of  determining  whether 
a  subsystem  or  component  of  the  system  should  be  developed  and 
built  by  the  system  developer  (in-house)  or  purchased  from  a 
source  outside  of  the  system  developer's  group  based  on 
criteria  such  as  cost,  time,  and  the  item's  technical 
performance.  This  does  not  covers  subsystems  and  components 
that  the  system  developer  has  decided  a  priori  not  to  perform 
a  make-or-buy  evaluation  based  on  the  development  philosophy, 
company  expertise  and  background,  or  security  classification. 
(For  example,  an  aircraft  integrator  without  experience  in 
developing  and  building  jet  engines  would  not  consider 
developing  and  building  them  in-house  for  a  new  aircraft.  The 
development  approach  would  be  to  obtain  them  from  sources  with 
specialties  in  jet  engines.) 

•  7-10  points  are  given  for  performing  make-or-buy  evaluations 
and  decisions  regarding  a  majority  of  a  system's  subsystems 
and  components  that  are  not  already  covered  by  a  reasonable 
a  priori  development  policy  reflecting  the  system 
developer's  in-house  capabilities  and  resources. 

•  4-6  points  are  given  for  performing  make-or-buy  evaluations 
and  decisions  regarding  some  of  the  system's  subsystems  and 
components  that  are  not  already  covered  by  a  reasonable 

a  priori  development  policy  reflecting  the  system 
developer's  in-house  capabilities  and  resources. 

•  0-3  points  is  given  for  little  or  no  consideration  of  the 


tradeoffs  between  developing  subsystems  and  components  in- 
house  and  purchasing  them  from  outside. 
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E.  Validation.  This  consists  of  the  validation  of  requirements  to 
ensure  they  are  consistent  with  the  customers '  needs  and  that 
a  real  world  solution  can  be  built  and  tested  to  prove  that  it 
satisfies  the  requirements. 

•  7-10  points  are  given  for  conducting  a  methodical  process 
involving  direct  interaction  with  the  customer  to  ensure 
the  system  requirements  are  consistent  with  the  true  needs 
of  the  customer,  and  determining  that  a  real  world  solution 
can  be  built  and  tested. 

•  4-6  points  are  given  for  ensuring  the  requirements  are 
consistent  with  the  true  needs  of  the  customer  through 
indirect  methods,  such  as  referencing  marketing  surveys,  and 
for  determining  that  a  real  world  solution  can  be  built  and 
tested. 

•  0-3  points  for  not  reviewing  requirements  with  regard  to 
actual  customer  desires  or  not  attempting  to  justify  that  a 
real  world  solution  exists. 


F.  Verification  and  integrated  testing.  This  figure  of  merit 
refers  to  the  determination  of  design  compliance  with 
performance  specifications  and  the  determination  of  as— built 
hardware  and  software  compliance  with  specification  and 
drawing  requirements.  This  includes  identifying  component- 
level  to  system— level  testing  in  a  test  and  evaluation  master 
plan  early  in  development.  It  also  refers  to  actually 
conducting  tests  and  inspections  in  a  complete  and  logical 
manner  of  components  and  subsystems  individually ,  then  of 
combinations  of  components  and  subsystems  together,  and 
finally  of  the  final  integrated  system. 
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•  7-10  points  are  given  for  a  complete  job  of  verifying  design 
and  hardware/ software  compliance  with  specifications  and 
drawings,  clearly  identifying  early  in  development  the 
planned  component,  subsystem,  and  system-level  testing  in  a 
comprehensive  test  plan,  and  then  actually  conducting 

the  testing. 

•  4-6  points  are  given  for  partially  verifying  design  and 
hardware/software  compliance  with  specifications  and 
drawings,  incompletely  identifying  early  in  development  the 
planned  component,  subsystem,  and  system-level  testing  in  a 
test  plan  or  series  of  plans,  and  then  eventually  conducting 
all  the  needed  testing. 

•  0-3  points  are  given  for  incomplete,  inadequate  attempts  to 
verify  design  and  hardware /software  compliance  with 
specifications  and  drawings,  not  planning  for  and 
identifying  early  in  development  the  progression  of  tests 
from  component-level  to  system-level  throughout  system 
development,  and  not  conducting  important  tests. 


G.  Configuration  management.  This  refers  to  establishing  and 
maintaining  the  status  of  the  design  configuration  and 
interfaces  and  as  defined  by  specifications  and  drawings 
(paper  or  digital),  controlling  changes  to  the  configuration, 
appropriately  controlling  changes  between  subsystems  and 
between  the  system  and  the  external  world,  and  maintaining 
the  traceability  of  the  configuration  as  it  changes. 

•  7-10  points  are  given  for  effectively  controlling  interface 
changes  within  a  formal  system,  maintaining  accurate  status 
of  the  design  configuration,  controlling  configuration 
changes  through  a  formal  system  of  review,  approval,  and 
update,  and  maintaining  the  traceability  of  the 
configuration . 

•  4—6  points  are  given  for  controlling  interface  changes 
within  a  formal  system  with  some  problems,  maintaining 
reasonably  accurate  status  of  the  design  configuration, 
controlling  configuration  changes  with  some  difficulty,  and 
maintaining  reasonable  traceability  of  the  configuration. 

•  0-3  points  are  given  for  inadequately  controlling  interface 
changes,  failing  to  maintain  accurate  status  of  the  design 


configuration,  inadequately  controlling  configuration 
changes,  and/or  not  maintaining  traceability  of  the 
configuration. 
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H.  Manufacturing  considerations.  This  is  defined  as  addressing 
and  giving  priority  to  appropriate  manufacturing 
considerations  early  in  system  development  with  the  objectives 
of  having  (1)  the  manufacturing  entity  have  adequate  time  to 
prepare  for  fabrication  and  production  and  (2)  the 
manufacturing  considerations  influence  the  design  of  the 
system  and  its  manufacturing  processes  to  reduce  fabrication 
costs . 

•  7-10  points  are  given  for  developing  manufacturing 
processes  concurrently  with  the  system  design,  and  allowing 
the  design  to  be  significantly  influenced  by  manufacturing 
considerations  to  improve  ease  and/or  cost  of  fabrication. 

•  4-6  points  are  given  for  involving  manufacturing  personnel 
early  in  development,  but  not  allowing  the  design  to 

be  significantly  influenced  by  them  to  improve  ease  and/or 
cost  of  fabrication. 

•  0-3  points  are  given  for  ignoring  or  placing  low  emphasis  on 
manufacturing  issues  during  system  design. 


I.  Systems  integration  and  technical  management.  This  function 
involves  bringing  subsystems  together  to  produce  the  desired 
results  and  ensure  that  the  subsystems  will  interact  to 
satisfy  the  customers'  needs.  This  involves: 

-  organizing  the  technical  effort;  identifying  how  the 
development  will  be  broken  out  and  managed 

-  integrating  activities  of  the  development  team; 
balancing  the  influence  of  all  required  design  specialties; 
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resolving  design  conflicts 

-  ensuring  compatibility  of  all  physical,  functional,  and 
program  interfaces 

-  assessing  and  managing  technical  risk 

•  7-10  points  are  given  for  effectively  bringing  subsystems 
together  through  a  well  organized  and  integrated 
technical  effort  that  balances  the  influence  of  all 
required  design  specialties. 

•  4-6  points  are  given  for  bringing  subsystems  together  with 
significant  difficulty,  due  to  inadequate  subsystem 
verification  or  problems  in  the  organization  of  the 
technical  activities  which  does  not  thoughtfully  balance  the 
influence  of  all  required  design  specialties. 

•  0-3  points  are  given  for  failing  to  effectively  bring 
subsystems  together. 


J.  Life  cycle  considerations.  This  refers  to  giving  priority  to 
long  term  issues,  such  as  supportability  (primarily 
maintainability,  reliability,  and  training)  and  life  cycle 
costs  consistent  with  appropriate  system  requirements  and 
objectives . 

•  7-10  points  are  given  for  fully  recognizing  and  addressing 
supportability  and  life  cycle  cost  requirements  and  issues 
early  in  development. 

•  4-6  points  are  given  for  recognizing  and  minimally 
addressing  supportability  and  life  cycle  cost  requirements 
and  issues  early  in  development. 

•  0-3  points  are  given  for  not  adequately  recognizing  and 
addressing  supportability  and  life  cycle  cost  requirements 
and  issues  early  in  development. 


K.  Program  management.  This  involves  the  planning,  tracking,  and 
coordination  of  activities  performed  by  all  elements  of  the 
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development  team  as  well  as  the  resolution  of  impediments  to 
program  progress.  Evidence  of  strong  program  management 
include  an  integrated  master  scheduling  system,  an  efficient 
cost  accounting  system  which  provides  management  visibility 
into  development  activities  in  a  timely  manner,  life  cycle 
cost  estimates,  risk  analyses,  and  the  conduct  of  regular 
program  reviews  involving  all  the  key  stakeholders. 

•  7-10  points  are  given  for  a  strong  program  management 
function  that  effectively  keeps  the  effort  on  track  through 
use  of  an  accurate  integrated  master  scheduling  system,  use 
of  a  cost  accounting  system  providing  timely  information, 
and  holding  regular  program  reviews  with  program 
stakeholders. 

•  4-6  points  are  given  for  a  program  management  function  of 
medium  strength  that  keeps  the  effort  only  moderately 

on  track  due  at  least  partly  to  problems  with  the 
management  support  systems  and  practices. 

•  0-3  points  are  given  for  a  weak  program  management  function 
that  does  not  keep  the  effort  on  track  and  has  deficient 
management  support  systems  and  practices. 


The  ratings  and  scores  are  provided  in  a  table  at  the  end  of  each 
case  study  similar  to  that  illustrated  in  Table  2.2. 


Table  2.2  Systems  Engineering  Fundamentals  scores. 


Figures  of  Merit 


REQUIREMENTS  DEVELOPMENT 


INCIPIENT  SYSTEM  DESIGN 


EVALUATING  ALTERNATIVE 
CONCEPTS  AND  DESIGNS 


MAKE-OR-BUY  DECISION 


VALIDATION 


VERIFICATION  AND  INTEGRATED 
TESTING 


CONFIGURATION  MANAGEMENT 


MANUFACTURING 

CONSIDERATIONS 


SYSTEMS  INTEGRATION  AND 
TECHNICAL  MANAGEMENT 


LIFE  CYCLE  CONSIDERATIONS 


PROGRAM  MANAGEMENT 


SYSTEMS  ENGINEERING 
FUNDAMENTALS  TOTAL 


Development  Environment 


A.  Emphasis  on  the  customer.  The  customer  requirements  are  the 
primary  design  drivers,  and  user  input  during  the  entire 
development  process  is  accepted  and  encouraged. 

•  7-10  points  are  given  for  a  high  emphasis  on  receiving  and 
positively  responding  to  direct  or  indirect  customer 
involvement  throughout  development. 

•  4-6  points  are  given  for  a  moderate  emphasis  on  receiving 
and  positively  responding  to  direct  or  indirect  customer 
involvement  throughout  development. 

•  0-3  points  are  given  for  little  or  no  emphasis  on  receiving 


or  positively  responding  to  inputs  from  the  customer 
throughout  development. 
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B.  Stability  of  requirements  and  configuration.  The  customer 
requirements  do  not  undergo  a  series  of  major  changes  after 
the  start  of  FSED,  and  the  system  configuration  does  not 
undergo  numerous  alterations,  whether  they  are  driven  by 
customer  requirements  changes  or  correction  of  design 
deficiencies. 

•  7-10  points  are  given  for  minor  changes  to  no  change  in 
customer  requirements  and/or  for  a  low  number  of  moderate 
configuration  changes  during  FSED. 

•  4-6  points  are  given  for  moderate  changes  in  customer 
requirements  and/or  for  a  low  number  of  major 
configuration  changes  during  FSED. 

•  0-3  points  are  given  for  major  changes  in  customer 
requirements  and/or  for  a  moderate  to  large  number  of  major 
configuration  changes  during  FSED. 


C.  Funding  and  workforce-level  stability.  This  means  that  the 
development  effort  follows  a  multi-year  budget  that  does 
not  change  significantly  each  year  from  original  plan. 
Furthermore,  the  workforce— level  throughout  FSED  follows  a 
long-term  plan. 

•  7-10  points  are  given  for  no  unplanned,  major  dips  or  spikes 
in  funding  amounts  and  workforce-levels  throughout  FSED. 

•  4-6  points  are  given  for  moderate  funding  and  workforce- 
level  deviations  from  long  term  plans  during  FSED. 

•  0-3  points  are  given  for  major  and  severe  funding  and 
workforce— level  deviations  from  long  term  plans  during  FSED. 
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D.  Strong  support  for  program.  This  is  defined  as  a 

development  effort  having  strong  general  support  within  its 
company  (referring  to  commercial  programs)  or  within  the 
government,  public  and  media  (referring  to  government 
programs)  during  FSED. 

•  7-10  points  are  given  for  strong  support  throughout  concept 
development  and  FSED  with  no  significant  controversy 
threatening  viability  of  the  effort. 

•  4-6  points  are  given  for  moderate  support  or  sharply 
contrasting  levels  of  support  during  concept  development  and 
FSED  due  to  significant  controversy  that  moderately 
threatens  the  viability  of  the  effort. 

•  0-3  points  are  given  for  little  or  no  support  resulting  from 
a  significant  controversy  during  concept  development  and 
FSED  that  seriously  threatens  the  viability  of  the  program. 


E.  Continuity  of  core  development  team.  This  means  that  a  core 
team  of  designers  and  managers  remaining  throughout 
development;  there  is  minimal  turnover  of  key 
personnel . 

•  7-10  points  are  given  for  there  being  only  little  or  no 
turnover  of  key  designers  and  managers  during  concept 
development  and  FSED. 

•  4-6  points  are  given  for  there  being  a  moderate  amount  of 
turnover  of  key  designers  and  managers  during  concept 
development  and  FSED. 

•  0-3  points  are  given  for  essentially  little  or  no  continuity 
of  key  designers  and  managers  during  concept  development  and 
FSED. 


F.  Stability  of  organizational  structure.  This  means  that  the 
system  development  organization  does  not  go  through  major 
reorganizations  during  the  development  of  the  system. 
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•  7-10  points  are  given  for  the  development  group  not  going 
through  a  major  reorganization  during  FSED . 

•  4-6  points  are  given  for  the  development  group  going  through 
a  moderate  reorganization  during  FSED. 

•  0-3  points  are  given  for  the  development  group  going  through 
a  major  reorganization  during  FSED. 

G.  Cooperation  among  all  stakeholders.  This  refers  to  the 
existence  of  positive,  non-confrontational  working 
relationships  among  the  team  members  and  between  the  team 
members  and  the  customer(s).  Furthermore,  there  are  no  major 
hidden  agendas  in  conflict  with  program  and  customer 
objectives . 

•  7-10  points  are  given  for  very  positive,  non-confrontational 
working  relationships  among  program  participants  during 
FSED . 

•  4-6  points  are  given  for  generally  positive  to  generally 
negative  working  relationships  existing  among  program 
participants  during  FSED. 

•  0—3  points  are  given  for  very  negative  working  relationships 
among  program  participants  that  severely  impede  or  stop 
program  progress  during  FSED. 

H.  Effective  communication  within  team.  This  is  defined  as  how 
well  members  of  the  development  team  (including  subcontractors 
and  customers )  communicate  and  coordinate  among  themselves .  It 
is  evidenced  by  adequate  mechanisms  for  communication,  such  as 
close  physical  proximity  of  the  workers  to  each  other  or  the 
existence  and  use  of  communication  mechanisms  such  as 
telephones,  fax  machines,  computer  aided  design/manufacturing 
systems,  and  computer  networks.  It  is  also  influenced  by  the 
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organizational  structure,  the  management  philosophy,  and 
cultural  factors  of  the  organization. 

•  7-10  points  are  given  for  effective  communication  and 
coordination  among  the  development  team  members  and 
between  them  and  the  customers. 

•  4-6  points  are  given  for  moderately  effective  communication 
and  coordination  among  the  development  team  members  and 
between  them  and  the  customers. 

•  0-3  points  are  given  for  ineffective  communication  and 
coordination  among  the  development  team  members  and  between 
them  and  the  customers  due  to  poor  circumstances  and 
mechanisms. 


I.  Flexibility  and  autonomy.  This  refers  to  the  ability  to 
implement  design  changes  guickly,  thereby  not  being 
significantly  hindered  by  organizational  or  procedural 
roadblocks.  Flexibility  and  autonomy  is  evidenced  by  fast  and 
efficient  design  change  mechanisms  and  by  the  absence  of 
bureaucratic  (government  or  corporate)  micromanagement  of 
development  activities. 

•  7-10  points  are  given  for  the  ability  to  make  decisions  and 
implement  design  changes  guickly  without  procedural 
roadblocks  or  bureaucratic  micromanagement. 

•  4-6  points  are  given  for  a  moderate  degree  of  procedural 
impediments  to  guick  decision  making  and  change 
implementation,  due  to  inflexible  and  inefficient 
procedures  and/or  micromanagement  of  development  activities 
by  corporate  or  government  bureaucracy . 

•  0-3  points  are  given  for  significant  procedural 
impediments  to  quick  decision  making  and  change 
implementation  and/or  a  large  amount  of  micromanagement  of 
development  activities  by  corporate  or  government 
bureaucracy. 
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J.  Workforce  expertise  and  experience.  This  means  that 

development  personnel  have  appropriate  skills  and  experience 
to  enable  development  success. 

•  7-10  points  are  given  for  most  development  team  members 
being  highly  skilled  and  experienced  in  appropriate  areas. 

•  4-6  points  are  given  for  a  moderate  number  of  development 
team  members  being  skilled  in  appropriate  areas,  but  lacking 
experience. 

•  0-3  points  are  given  for  a  severe  lack  of  appropriate  skills 
and  expertise  among  development  team. 


K.  Accountability  for  system  performance.  This  means  that  the 
system  developer  is  held  accountable  to  the  customer  for  the 
performance  of  the  resulting  system.  Such  a  situation  is 
evidenced  by  the  existence  of  mechanisms  such  as  warranties 
and  product  liability  laws.  Another  factor  is  the  importance 
of  reputation  to  the  developer,  as  well  as  the  significance  of 
reputation  to  future  efforts.  Accountability  for  system 
performance  also  means  that  the  system  developer  is 
responsible  to  its  funding  source  for  the  way  the  effort  is 
conducted,  and  this  is  evidenced  by  award  fee  and  incentive 
fee  structures  for  cost  type  government  contracts,  bonus 
programs  for  commercial  efforts,  and  mechanisms  for  quickly 
replacing  personnel  if  progress  is  not  satisfactory. 

•  7-10  points  are  given  for  strong  warranties  on  key 
performance  parameters  and  defective  parts,  for  award  and 
bonus  fee  mechanisms,  for  product  liability  laws,  and/or 
strong  motivation  to  maintain  company  reputation. 

•  4-6  points  are  given  for  warranties  covering  the  replacement 
of  defective  parts  and  excluding  some  key  performance 
parameters,  in  addition  to  limited  award  fee  or  bonus 


49 


mechanisms  and  product  liability  laws. 

•  0-3  points  for  no  warranty  or  extremely  limited  warranty 
covering  replacement  of  defective  parts  and  no  other 
significant  accountability  mechanisms  in  place. 


The  ratings  and  scores  are  provided  in  a  table  at  the  end  of  each 
case  study  similar  to  that  illustrated  in  Table  2.3. 


Table  2.3  Development  Environment  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

EMPHASIS  ON  THE  CUSTOMER 

0-10 

1 

STABILITY  OF  REQUIREMENTS 

AND  CONFIGURATION 

0-10 

1 

FUNDING  AND  WORKFORCE-LEVEL 
STABILITY 

0-10 

1 

STRONG  SUPPORT  FOR  PROGRAM 

0-10 

1 

CONTINUITY  OF  CORE 
DEVELOPMENT  TEAM 

0-10 

1 

STABILITY  OF  ORGANIZATIONAL 
STRUCTURE 

0-10 

1 

COOPERATION  AMONG  ALL 
STAKEHOLDERS 

0-10 

1 

EFFECTIVE  COMMUNICATION 
WITHIN  TEAM 

0-10 

1 

FLEXIBILITY  AND  AUTONOMY 

0-10 

1 

WORKFORCE  EXPERTISE  AND 
EXPERIENCE 

0-10 

1 

ACCOUNTABILITY  FOR  SYSTEM 
PERFORMANCE 

0-10 

1 

DEVELOPMENT  ENVIRONMENT 
TOTAL 


0-110 
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Design  Difficulty 

A.  Design  type  reflects  whether  feasible  solutions  exist  and  how 
much  original  thought  goes  into  the  project. 

•  14  or  15  points  are  given  for  a  breakthrough  design  effort. 

•  7-13  points  are  given  for  original  innovative  design. 

•  0-6  points  are  given  for  continuous  improvement. 

B.  Complexity  of  the  knowledge  needed  to  create  the  design  is 
determined  based  on  an  estimate  of  the  number  and  availability 
of  the  people  with  the  necessary  knowledge  to  do  the  design. 

•  9  or  10  points  are  given  for  undiscovered  knowledge  that  can 
found  only  by  specialists. 

•  6-8  points  are  given  for  complex  knowledge  held  by  few 
people . 

•  3-5  points  are  given  for  complex  knowledge  held  by  a 
sufficient  pool  of  people 

•  0-2  points  are  given  for  common  knowledge  held  by  many 
people . 

C.  The  number  of  steps  needed  to  complete  the  design  is  defined 
as  the  number  of  discrete  steps  needed  to  design  the  system. 

It  is  related  to  the  number  of  major  components  or  major 
process  steps  that  are  needed  to  assemble  the  system. 

•  9  or  10  points  are  given  for  systems  with  greater  than 
10,000  steps  or  components. 

•  5-8  points  are  given  for  systems  with  more  than  500  but  less 
than  10,000  steps  or  components. 

•  3  or  4  points  are  given  for  systems  up  to  500  steps  or 
components . 
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•  0-2  points  are  given  for  any  system  with  fewer  than  50 
steps  or  components. 


D.  Quality  implementation  effort  refers  to  the  extent  to  which 
the  company  designing  and  building  the  system  follows  advanced 
quality  programs  and  practices,  such  as  total  quality 
management  (TQM),  ISO-9000,  Baldridge  Award  criteria,  Six 
Sigma,  Taguchi  methods,  and  Quality  Function  Deployment  (QFD). 

•  7-10  points  are  given  for  a  system  whose  developer  places 
high  emphasis  on  implementing  or  continuing  quality-related 
programs  and  techniques  on  the  system  development  effort. 

•  4-6  points  are  given  for  a  system  whose  developer  places 
medium  emphasis  on  implementing  or  continuing  quality- 
related  programs  and  techniques  on  the  system  development 
effort . 

•  0-3  points  are  given  for  a  system  whose  developer  places 
little  or  no  emphasis  on  implementing  or  continuing  quality- 
related  programs  and  techniques. 


E.  Manufacturing  operations  implementation  effort  addresses  the 

combination  of  the  complexity  of  the  fabrication  processes  and 

the  quantity  of  the  items  produced.  For  example,  the 

manufacturing  arrangement  to  produce  one  or  a  few  large, 

complex  systems  can  be  as  extensive  as  those  established  to 

mass  produce  small,  less  complex  systems.  Quantity  is 

normalized  between  different  items  by  partly  basing  the 

measure  on  the  extent  of  national  market  share  met  by  the 

output  of  a  system's  manufacturing  operations. 

•  5  points  are  given  for  highly  complex  manufacturing 
operations  that  are  designed  to  produce  systems  in 
quantities  to  meet  a  large  national  market  share. 
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•  4  points  are  given  for: 

(1)  highly  complex  manufacturing  operations  that  are 
designed  to  produce  systems  in  quantities  to  meet  a 
moderate  national  market  share,  or 

(2)  manufacturing  operations  of  moderate  complexity  that  are 
designed  to  produced  systems  in  quantities  to  meet  a 
large  national  market  share. 

•  3  points  are  given  for: 

(1)  highly  complex  manufacturing  operations  that  are 
designed  to  produce  systems  in  quantities  to  meet  a 
small  national  market  share,  or 

(2)  manufacturing  operations  of  moderate  complexity  that  are 
designed  to  produce  systems  in  quantities  to  meet  a 
moderate  national  market  share,  or 

( 3 )  manufacturing  operations  of  low  complexity  that  are 
designed  to  produce  systems  to  meet  a  large  national 
market  share . 

•  2  points  are  given  for: 

(1)  manufacturing  operations  of  moderate  complexity  that  are 
designed  to  produce  systems  in  quantities  to  meet  a 
small  national  market  share,  or 

(2)  manufacturing  operations  of  low  complexity  that  are 
designed  to  produce  systems  in  quantities  to  meet  a 
moderate  national  market  share. 

•  1  point  is  given  for  manufacturing  operations  of  low 
complexity  that  are  designed  to  produce  systems  in 
quantities  (of  greater  than  one  unit)  to  meet  a  small 
national  market  share. 

•  0  points  are  given  for  manufacturing  operations  of  low 
complexity  that  are  designed  to  produce  only  one  system, 
meeting  only  a  small  national  market  share. 


F.  Selling  price  constraint  is  defined  as  the  degree  to  which  the 
system  design  is  driven  and  constrained  by  unit  sales  price 
requirements  or  goals.  These  requirements  and  goals  are  set 
based  on  the  competition  level  in  the  market.  In  general,  the 
greater  the  competition,  the  greater  the  constraint.  The  sales 
price  requirements  and  goals  for  government  systems  can  be 
determined  by  design-to-cost  requirements  or  goals  as  well  as 
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by  the  existence  of  equivalent  commercial  systems  already  on 
the  market. 

•  4-5  points  are  given  for  very  challenging  unit  sales  price 
requirements  or  goals  driven  by  a  highly  competitive  market. 

•  2-3  points  are  given  for  moderately  challenging  unit  sales 
price  requirements  or  goals  driven  by  a  moderately 
competitive  market. 

•  0-1  points  are  given  for  little  or  no  challenge  to  meet  unit 
sales  price  requirements  or  goals  due  to  lack  of  competition 
or  no  unit  sales  price  requirements  or  goals  exist. 


The  scores  are  provided  in  a  table  at  the  end  of  each  case  study 
similar  to  that  illustrated  in  Table  2.4. 


Table  2.4  -  Design  Difficulty  scores. 


Element 

Range 

Score 

TYPE 

0-15 

KNOWLEDGE  COMPLEXITY 

0-10 

STEPS 

0-10 

QUALITY  IMPLEMENTATION 
EFFORT 

0-10 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

SELLING  PRICE  CONSTRAINT 

0-5 

DESIGN  DIFFICULTY  TOTAL 

0-55 

Resources 


A.  Cost  is  the  amount  needed  to  pay  for  development,  including 
salaries,  utilities,  supplies,  and  materials,  through  the 
first  production  unit.  This  is  not  in  absolute  dollars,  but  in 
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terms  of  the  payer's  ability  to  pay.  For  example,  it  is  easy 
for  a  rich  man  to  afford  a  VAX  computer,  but  not  a  person  with 
an  average  salary.  Combine  1,000  average  salaries  and  these 
people  can  afford  a  VAX. 

•  14  or  15  points  are  given  for  massively  expensive  systems 
requiring  major  sacrifices. 

•  9-13  points  are  given  for  very  expensive  systems  that  are 
developed  rarely. 

•  3-8  points  are  given  for  moderately  expensive  systems. 

•  0-2  points  are  given  for  affordable  systems. 

B.  The  time  score  is  for  time  spent  from  the  beginning  of  the 
effort  to  define  the  customer's  needs  through  the  first 
production  unit. 

•  10  points  are  given  for  more  than  eight  years. 

•  8  or  9  points  are  given  for  up  to  eight  years. 

•  4-7  points  are  given  for  up  to  five  years. 

•  3  points  are  given  for  a  year. 

•  2  points  are  given  for  three  months  to  around  six  months. 

•  1  point  is  given  for  a  month  to  less  than  three  months. 

•  0  points  are  given  for  less  than  a  month. 

C.  Infrastructure  required  to  achieve  the  design  is  also  hard  to 
quantify.  Infrastructure  is  described  as  physical  resources 
needed  for  construction  (including  machine  tools,  process 
shops,  and  assembly  workstations),  transportation, 
communication,  utilities,  laws  and  legal  protections, 
skilled  managers,  and  the  education  system  available. 
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Infrastructure  must  be  judged  in  regard  to  the  designer's 
ability  to  get  and  use  the  infrastructure  over  the  needed 
design  time. 

•  9  or  10  points  are  given  for  a  massive  infrastructure 
requiring  major  portions  of  the  available  labor  force  and 
the  available  equipment. 

•  6-8  points  are  given  for  large  complex  infrastructures 
requiring  large  portions  of  the  cost  of  the  entire  project 

•  3-5  points  are  given  for  moderate  infrastructures  requiring 
people  on  the  project  to  support  it. 

•  0-2  points  are  given  if  it  is  a  common,  low  cost 
infrastructure  (e.g.  clean  tap  water  in  the  U.S.) 


The  scores  are  given  in  a  table  at  the  end  of  each  case  study- 
similar  to  that  illustrated  in  Table  2.5. 


Table  2.5  Resources  scores. 


Element 

Range 

Score 

COST 

0-15 

TIME 

0-10 

INFRASTRUCTURE 

0-10 

RESOURCES  TOTAL 

0-35 

Explanation  of  Scoring 

Each  case  in  Chapter  3  is  evaluated  using  the  rating  and  scoring 
methodology  above.  After  ratings  are  assigned  to  the  individual 
figures  of  merit  and  elements,  they  are  totaled  for  each 
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development  effort  characteristic  (performance ,  systems 
engineering  fundamentals ,  development  environment ,  design 
difficulty ,  and  resources) . 

The  final  scores  reported  in  this  thesis  were  determined  by  the 
author  using  inputs  from  nine  engineers  who  individually  read  and 
rated  the  case  studies.  The  final  scores,  however,  are  not  the 
averages  of  those  from  the  different  individuals.  These  final 
case  study  scores  are  compiled  and  analyzed  in  Chapter  4. 

The  case  study  evaluations  by  the  nine  engineers  were  primarily 
used  to  help  perform  an  initial  validation  of  the  evaluation 
methodology  by  identify  weaknesses  in  the  case  studies  and  the 
ratings  criteria.  Based  on  the  mean  and  variability  of  the 
group's  scores,  the  author  improved  passages  in  case  studies, 
strengthened  the  wording  and  content  of  some  rating  criteria,  and 
modified  inappropriate  scores.  The  validation  data  is  presented 
and  discussed  in  Chapter  5. 
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CHAPTER  3 
CASE  STUDIES 


The  sources  of  information  used  in  the  preparation  of  the 
following  six  case  studies  included  periodicals,  questionnaires, 
interviews,  reports,  books,  and  brochures.  References  are 
presented  at  the  end  of  each  case.  A  sample  questionnaire  is 
included  in  the  Appendix. 
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3.1  CASE  STUDY:  BOEING  777  COMMERCIAL  TRANSPORT 


The  Boeing  777  is  the  world's  largest  twin-engine  commercial 
transport  currently  being  developed  for  long  distance  commercial 
passenger  and  cargo  travel .  It  is  intended  to  fill  a  market  and 
product  gap  between  Boeing's  767-300  and  the  747-400  jumbo  jet, 
and  it  will  compete  directly  against  the  McDonnell  Douglas  MD— 11 
trijet  and  the  four-engine  Airbus  Industrie  A330/340  family.  The 
777  is  unique  in  that  it  was  developed  in  a  significantly 
different  manner  than  the  earlier  Boeing  aircraft,  and  even  its 
competitors.  Three  characteristics  of  this  development  were  (1) 
multi-disciplinary  teams  working  together  to  concurrently  design 
the  aircraft  and  its  production  processes,  (2)  an  unprecedented 
participation  of  the  customer  airlines  in  the  development 
process,  and  (3)  the  design  and  integration  work  being  done 
almost  entirely  on  computer.  All  of  these  innovations  were 
intended  to  produce  an  aircraft  that  was  free  of  problems  and 
ready  for  full  service  when  delivered  to  the  customer. 

Development  History,  Design,  and  Performance  Outcomes 

The  Boeing  Company  officially  launched  the  777  development 
program  on  October  29,  1990,  when  its  corporate  board  of 
directors  gave  approval  to  take  what  was  then  a  concept 
development  effort  into  full-scale  engineering  development  (FSED) 
and  production.  Despite  some  problems  with  engine  development. 
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the  effort  progressed  according  to  plan.  The  first  flight  took 
place  according  to  original  schedule  on  June  12,  1994,  and  flight 
testing  to  certify  the  three  different  engines  available  to 
customers  is  expected  to  run  through  March  1996.  Boeing  made  its 
first  777  delivery  to  United  Airlines  on  schedule  in  May  1995. 

( See  Figure  3.1.) 

As  of  late  1994,  there  were  147  firm  orders  and  108  options  for 
the  $125  million  transports.  While  the  primary  financing  has  come 
from  Boeing,  the  Japanese  companies  that  are  fabricating  large 
portions  of  the  fuselage  planned  to  put  up  from  8-10  percent  of 
the  estimated  $4  to  $5  billion  start— up  costs.  [O'Lone,  1990] 

Figure  3.1  777  development  schedule 

1986  1987  1988  1989  1990  1991  1992  1993  1994  1995 

Reqmts  &  Concept  Dev. 

1st  Flight  ^ 

Flight  Testing  (Certification) 

1st  Production  Delivery  ^ 


The  777  can  be  regarded  as  an  original  design.  However,  it  can 
also  be  described  as  a  major  redesign  that  is  evolutionary  in 
nature  since  the  design  is  based  primarily  on  experience  gained 
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from  the  Boeing  757  and  767  introduced  in  the  early  1980s  and  the 
latest  version  of  the  747  introduced  in  1989.  The  777  has  the 
appearance  of  an  enlarged  767,  and  several  major  items  were  taken 
from  the  757,  767,  and  747-400.  For  example,  the  nose  structure 
of  the  777  is  the  same  as  the  767.  The  cockpit  design  was  derived 
from  the  747—400.  The  engine  attachment  is  a  scaled  version  of 
the  design  used  on  the  757.  Also,  one  of  the  three  engines  that 
will  be  certified  with  the  777,  the  Pratt  and  Whitney  PW4074,  is 
a  larger  derivative  of  the  existing  PW4060  used  on  the  767.  In 
addition  to  using  subsystems  and  components  from  earlier 
aircraft,  the  777  utilizes  a  significant  amount  of  advanced 
technologies  both  inside  and  out  to  meet  challenging  performance 
and  design  goals. 

The  primary  777  goal  was  to  develop  a  large  twin-engine  transport 
that  could  beat  the  fuel  consumption  performance  of  a  three-and 
four-engine  aircraft.  Boeing  originally  designed  the  777  to  be  6- 
8  percent  more  fuel  efficient  than  the  MD— 11  and  the  A330/A340. 
[Dornheim,  1991]  To  accomplish  this  required  the  use  of  advanced 
materials,  lightweight  flight  controls,  and  an  improved 
aerodynamic  design.  For  the  first  time  on  a  Boeing  commercial 
transport,  some  of  the  primary  structure  is  made  of  lightweight 
composites.  Specifically,  the  horizontal  and  vertical  tail 
structural  boxes  are  made  of  an  advanced  graphite  epoxy  material, 
and  composites  are  used  in  the  spoilers  and  flaps.  Overall 
composite  content  is  about  nine  percent  by  weight.  Boeing  also 
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used  a  new  aluminum  alloy  for  the  upper  wing  skin  which  is  a 
stronger  refinement  of  materials  used  on  the  company's  three  most 
recent  transports.  Lower  weight  was  also  achieved  using  fly-by- 
wire  flight  controls,  thereby  eliminating  the  need  for  heavy 
hydraulics.  The  use  of  electronic  flight  controls,  which  have 
been  used  in  military  aircraft  since  the  middle  1970's,  was  a 
first  for  a  Boeing  commercial  transport.  These  approaches  to  keep 
weight  down  have  been  very  effective,  yet  the  777  was  expected  to 
be  several  percent  overweight.  [O' Lone,  1991] 

In  addition  to  reducing  weight,  the  777  designers  intended  to 
achieve  fuel  consumption  goals  by  improving  aerodynamic 
efficiency.  According  to  Boeing,  the  777  wing  is  the  most 
aerodynamical ly  efficient  airfoil  ever  developed  for  a  subsonic 
commercial  aircraft.  [Boeing,  ca  1994]  The  777  development  team 
also  faced  a  major  aerodynamic  challenge  of  fitting  the  large 
engine  nacelle  onto  the  wing  with  the  least  drag  penalty .  To 
further  reduce  drag,  Boeing  designed  parts  of  the  airplane  to 
exceptionally  close  tolerances  for  a  tight  fit.  [Air  &  Space, 
1994]  By  using  the  advanced  supercomputer  simulation  tool  of 
computational  fluid  dynamics  (CFD)  and  extensive  wind  tunnel 
testing  to  validate  the  CFD  models,  the  777  designers  were  able 
to  produce  an  overall  design  that  met  its  aerodynamic  performance 
goals. 
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The  777  team  had  another  challenge  developing  the  propulsion 
units.  Since  this  large  airplane  is  designed  to  fly  with  only  two 
engines,  each  must  be  very  large  and  powerful.  In  fact,  the 
diameter  of  each  of  the  three  different  candidate  engines  is 
about  the  same  as  the  fuselage  diameter  of  the  737.  The  three 
companies  which  each  developed  an  engine  for  the  777  are  Pratt  & 
Whitney,  General  Electric,  and  Rolls  Royce.  Of  these,  only  Pratt 
&  Whitney  offered  a  unit  that  was  derived  directly  from  an 
existing  engine  design,  thereby  avoiding  considerable 
development.  Despite  being  a  derivative  design,  the  Pratt  & 
Whitney  unit  did  experience  problems  during  development  testing. 
Resolving  these  problems  has  been  crucial  to  Boeing,  because 
sales  of  the  777s  are  dependent  on  special  early  Federal  Aviation 
Administration  ( FAA)  certification  called  extended  twin-engine 
operations  overwater  (ETOPS).  This  certification  allows  twin 
engine  aircraft  to  fly  long  distance  flights  over  water.  All 
prior  twin  engine  passenger  jets  that  fly  intercontinental 
distances  over  oceans  have  required  several  years  of  overland 
experience  to  develop  a  reliability  history  to  justify  ETOPS 
certification.  Without  ETOPS  certification  at  time  of  the  first 
777  deliveries,  the  aircraft  cannot  be  used  for  many  of  the 
routes  for  which  they  are  being  purchased.  ETOPS  was  granted 
before  the  first  scheduled  passenger  flight. 

Another  key  feature  that  posed  challenges  to  designers  was  the 
highly  integrated  and  automated  advanced  avionics,  enabling 
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operation  by  a  crew  of  only  two  pilots  instead  of  a  usual  cockpit 
crew  of  three.  The  resulting  Airplane  Information  Management 
System  (AIMS)  from  Honeywell  incorporates  into  a  single  system 
the  flight  management,  airplane  condition  monitoring,  central 
maintenance  and  digital  link  communication  functions,  which  in 
earlier  aircraft  were  performed  separately.  Despite  the  high 
degree  of  integration,  the  cockpit  has  been  designed  with  the 
growth  potential  to  accommodate  future  needs  for  additional 
information.  [Scott,  1991] 

Design  flexibility  has  also  been  incorporated  throughout  the  rest 
of  the  777  so  that  it  can  meet  a  variety  of  customer 
requirements.  For  example,  the  initial  version  will  carry  up  to 
375  passengers  a  distance  of  over  4,560  miles  (3,970  nautical 
miles).  Boeing  plans  to  evolve  the  design  into  configurations  for 
carrying  more  than  300  passenger  nearly  7,250  miles  (6,300 
nautical  miles)  and  about  400  passengers  over  7,000  miles.  The 
floor  was  strengthened  to  accommodate  the  weight  of  future 
increases  in  passengers  and  flight  amenities.  Furthermore,  zones 
of  flexibility  have  been  designed  into  the  cabin  areas  as 
specified  by  the  airlines.  Within  these  zones,  which  have  been 
pre— engineered  to  accommodate  wiring,  plumbing,  and  attachment 
fixtures,  the  galleys  and  lavatories  can  be  positioned  anywhere 
in  one-inch  increments.  Furthermore,  passenger  service  units  and 
overhead  stowage  compartments  are  designed  for  quick  removal 
without  disturbing  ceiling  panels,  air  conditioning  ducts,  or 
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support  structures.  A  typical  777  cabin  configuration  change  is 
expected  to  take  as  little  as  72  hours,  while  such  a  change  might 
take  two  to  three  weeks  on  existing  aircraft.  [Boeing,  ca  1994] 

By  any  measure,  the  777  is  a  large  and  complex  system.  It  will 
have  about  132,500  engineered,  unique  parts.  By  including  rivets, 
bolts,  and  other  fasteners  in  the  count,  the  airplane  will  have 
more  than  three  million.  [Boeing,  ca  1994]  Boeing's  job  is  to 
ensure  that  those  many  parts  work  together  in  a  manner  that  will 
satisfy  its  customers.  Customer  satisfaction  with  the  777  cannot 
be  directly  measured  yet  since  the  first  aircraft  had  just  been 
delivered  when  this  case  study  was  completed.  However,  the 
approach  with  which  Boeing  developed  the  777  suggests  that  the 
airline  customers  will  be  satisfied  with  what  they  receive. 

Systems  Engineering  Fundamentals 

Boeing  is  the  world's  largest  developer  and  manufacturer  of 
commercial  passenger  jet  aircraft.  Since  the  late  1950s,  the 
company  has  developed  the  707,  727,  737,  747,  757,  and  767  series 
of  passenger  jets.  Over  those  years,  Boeing  had  developed  and 
followed  its  version  of  a  traditional  approach  to  developing 
aircraft.  That  is,  design  engineers  designing  the  aircraft 
separately  and  then  giving  the  drawings  and  specifications  to 
manufacturing  for  fabrication  and  to  maintenance  personnel  for 
establishing  maintenance  procedures.  This  approach  normally  led 
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to  significant  changes  as  the  manufacturing  engineers  discovered 
design  inconsistencies  and  unproducible  component  configurations. 
It  also  resulted  in  higher  life  cycle  costs,  since  the  designs 
did  not  always  emphasize  ease  of  repair  and  reduced  need  for 
maintenance.  With  the  777,  Boeing  decided  on  a  radically  new 
development  approach  that  focused  on  the  customer  and  placed 
great  emphasis  on  ensuring  that  the  development  activities  were 
done  right  the  first  time. 

In  late  1986,  Boeing  started  identifying  requirements  for  a  large 
passenger  airplane  with  the  capacity  between  the  twin-engine  767- 
300  and  the  four-engine  747-400  jumbo  jet.  This  investigation 
into  what  Boeing  then  called  the  767-X  configuration  was 
initiated  as  a  result  of  interest  expressed  by  airlines  for  a 
medium-  to  long-distance  Boeing-produced  aircraft  in  the  300-400 
passenger  capacity  range.  Boeing  entered  into  a  dialogue  with 
interested  airlines  and  potential  subcontractors  to  define  and 
refine  requirements.  Trade  studies  were  conducted  throughout  this 
time,  culminating  in  a  system  design  concept  with  detailed 
requirements  allocated  to  lower  level  elements  of  the  aircraft. 

By  the  end  of  this  period,  a  group  of  national  and  international 
subcontractors  were  ready  to  develop  and  fabricate  many  of  the 
subsystems.  This  four-year  requirements  generation,  requirements 
validation,  concept  development,  and  make— or— buy  decision  effort 
culminated  with  the  launching  of  FSED . 
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Boeing's  development  and  systems  engineering  approach  is 
highlighted  by  early  participation  of  all  disciplines  as  well  as 
by  unprecedented  input  by  customer  airlines.  The  basic 
organizational  entity  responsible  for  designing  and  building  a 
portion  of  the  aircraft  is  the  design/build  team,  with  its 
membership  comprised  of  design  engineering,  manufacturing, 
specialty  engineering,  and  non-technical  representatives,  as  well 
as  representatives  of  the  subcontractors  and  suppliers.  The 
objective  of  the  approach  is  to  cut  development  costs  by  reducing 
the  amount  of  downstream  design  changes  and  resultant  rework,  the 
predominant  aircraft  development  cost  drivers.  [O'Lone,  1991] 

On  the  777  program,  the  teams  have  been  organized  around  parts  of 
the  aircraft  rather  than  functions .  At  the  top  are  thirty 
integration-level  teams  representing  the  largest  aircraft 
sections,  and  each  has  had  the  responsibility  to  maintain  the 
interfaces  of  its  component  parts  to  the  other  twenty-nine 
sections.  Various  levels  of  subsystem  and  component  sub-teams 
operate  below.  During  the  height  of  development,  up  to  238  of 
these  cross— functional  design/build  teams  worked  on  the  effort  at 
one  time.  [Boeing,  1994] 

Subcontractors  and  suppliers  have  had  unusually  close  working 
relationships  with  Boeing  under  this  structure.  [O'Lone,  1991] 

For  example,  in  cases  where  the  subcontractor  was  going  to  build 
the  hardware,  that  company's  representative  took  the  lead 
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manufacturing  role  on  the  design/build  team  instead  of  a  Boeing 
employee.  The  airline  companies  that  placed  the  first  orders  have 
also  taken  part  in  the  design/build  process  by  participating  in 
integration— level  design/build  meetings,  preliminary  design 
reviews,  and  critical  design  reviews.  This  continual  involvement 
by  the  customers  and  their  active  involvement  in  defining  the 
design  helped  to  validate  the  requirements  that  the  airlines  had 
established  before  the  program  started. 

A  crucial  element  of  Boeing's  development  strategy  was  designing 
the  777  on  an  advanced  computer  aided  design  (CAD)  system  instead 
of  using  paper  drawings.  This  tool,  called  CATIA,  is  a  three- 
dimensional  computer  system  with  solid  modeling  capability  that 
can  produce  a  virtual  prototype.  That  is,  the  parts  can  be 
separately  drawn  in  three  dimensions,  joined  together  in  the 
computer,  and  visually  displayed  on  the  screen.  This  enabled  each 
team  to  create  their  designs  and  compare  them  with  the  other 
teams'  work  to  check  for  interference  between  parts.  With  over 
2,000  computer  terminals  linked  together  and  available  to  the 
development  team,  CATIA  improved  the  speed  of  determining  and 
communicating  the  status  of  interfaces  and  configurations,  and  it 
greatly  increased  the  effectiveness  of  configuration  control 
activities. 

Engineers  have  also  been  able  to  use  CATIA  to  analyze  weights, 
balance,  and  stress  on  the  different  parts,  allowing  for  quick 
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evaluation  and  refinement  of  alternative  designs.  This  virtual 
prototyping  not  only  eliminated  the  need  for  most  paper  drawings, 
but  also  the  need  for  and  cost  of  most  physical  mockups  and  full- 
scale  prototypes.  The  most  significant  example  is  that  the  first 
777  produced  and  flown  was  a  production  configuration  aircraft 
instead  of  an  engineering  prototype. 

In  addition  to  design,  analysis,  and  verification  activities,  the 
CATIA  digital  design  system  has  had  a  large  impact  on 
manufacturing  operations  as  well .  CATIA  design  information  was 
combined  with  computer  controlled  machining  techniques,  and  it 
has  resulted  in  a  significant  improvement  in  dimensional 
accuracy.  [O'Lone,  1992]  CATIA  has  also  effected  the  replacement 
of  plaster  master  models  with  digital  data,  greatly  improving  the 
precision  and  efficiency  in  building  manufacturing  tools. 
Additionally,  Boeing  has  implemented  a  paperless  manufacturing 
floor  where  the  assembly  and  installation  shop  floor  control 
system  is  computer  based  and  enhanced  by  graphical  instructions 
with  links  to  the  CATIA  design  database.  Due  to  the  digital 
design  and  manufacturing  system,  Boeing  reports  a  50  percent 
reduction  of  rework  and  factory  floor  changes  compared  to  the 
767.  [Proctor,  1993] 

Boeing  planned  a  54  month  development  schedule,  which  is  somewhat 
greater  than  its  traditional  48  months  for  a  new  commercial 
aircraft.  [O'Lone,  1989]  This  was  to  allow  more  time  for  Boeing 
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to  work  out  problems  before  aircraft  delivery.  In  essence,  Boeing 
is  taking  the  aircraft  through  a  break-in  period  to  identify  and 
correct  annoying  problems  instead  of  having  the  airlines 
experience  them.  This  "service-ready"  approach  was  the  guiding 
philosophy  of  the  development  program,  and  it  drove  the  Boeing 
team  to  interact  closely  with  the  customer  during  development,  to 
conduct  extensive  integrated  ground  testing,  to  perform 
additional  flight  testing,  and  to  address  life  cycle 
support abi 1 i ty  issues  early.  This  approach  was  motivated  by  the 
significant  difficulties  experienced  by  airlines  when  they 
received  the  initial  deliveries  of  the  747—400  in  1989. 

As  mentioned  above,  one  of  the  strategies  for  achieving  the 
service  ready  goals  has  been  extensive  integrated  ground  testing. 
This  has  been  a  key  element  in  identifying  and  solving 
development  problems  as  early  as  possible  in  order  to  attain 
reliability  goals.  Boeing  invested  $370  million  in  a  new  test 
facility  called  the  Integrated  Aircraft  Systems  Laboratory  (IASL) 
to  accomplish  this.  The  IASL  tests  individual  parts, 
subassemblies,  and  integrated  aircraft  systems  both  on  the  static 
bench  and  under  simulated  flight  conditions.  During  advanced 
testing,  multiple  hardware  systems  are  integrated  and  operated 
"in-the-loop"  with  the  computer  simulations.  In  addition  to 
ground  testing  of  subsystems,  Boeing  demonstrated  the  performance 
of  its  fly-by-wire  design  and  cockpit  avionics  on  its  flying 
avionics  testbed,  a  modified  757,  prior  to  the  first  777  flight. 
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Extensive  flight  testing  was  another  strategy  being  used  to 
achieve  "service  ready"  goals.  Using  nine  777  aircraft,  Boeing 
accumulated  about  twice  as  much  flight  test  time  before 
delivering  the  first  aircraft  to  customers  as  it  normally  would. 
[Dornheim,  1994]  While  Boeing  has  had  to  conduct  extensive 
flight  testing  to  certify  performance  of  the  777,  about  60 
percent  of  the  overall  test  time  was  aimed  solely  at  meeting 
service  ready  goals  and  is  not  required  for  FAA  certification. 
[Dornheim,  1994] 

Placing  high  emphasis  on  life  cycle  considerations  early  in 
development  was  another  important  Boeing  strategy .  Much  of  the 
input  into  design  decisions  affecting  supportability  issues  came 
from  the  airlines.  Most  of  these  changes  focused  on  improving 
aircraft  reliability  and  maintainability.  Specifically,  lowering 
maintenance  costs  was  a  design  goal,  because  maintenance  labor 
expenses  consume  up  to  35  percent  of  revenues  at  some  airlines. 
[Proctor,  1994b]  An  example  of  development  team  supportability 
emphasis  is  the  Onboard  Maintenance  System  (OMS).  The  OMS 
operates  as  part  of  the  AIMS  discussed  earlier,  and  it 
automatically  records  data  of  interest  to  maintenance  personnel 
at  the  aircraft's  destination,  thereby  reducing  the  time  to 
determine  corrective  actions.  Another  example  is  that  shop 
maintenance  personnel  from  four  airlines  evaluated  proposed 
procedures  for  speed  and  degree  of  difficulty  as  soon  as  draft 
versions  were  issued.  The  result  is  that  final  maintenance 
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manuals  were  ready  for  the  fourth  flight  test  aircraft  instead  of 
after  initial  customer  deliveries  as  is  the  norm  for  Boeing 
aircraft.  Also,  due  to  the  close  coordination  between  Boeing  and 
the  airline  maintenance  department,  United  Airlines  will  for  the 
first  time  directly  use  Boeing  maintenance  manuals  instead  of 
redeveloping  them  first.  [Proctor,  1994a] 

Another  Boeing  supportability  objective  was  to  have  pilot 
training  simulators  in  operation  prior  to  delivery  of  the  first 
aircraft.  CAE  Electronics  Inc.  worked  closely  with  Boeing  to 
develop  111  flight  simulators  in  parallel  with  the  aircraft,  and 
a  partial  simulator  was  available  to  support  training  for  the 
first  flight. 

As  a  result  of  the  maintenance  improvements  as  well  as  the  fuel 
efficiency  advances,  Boeing  estimates  that  the  111  will  cost  10 
percent  less  to  operate  than  the  four  engine  A340  and  about  8 
percent  less  than  the  three  engine  MD— 11.  [Holusha,  1991] 

To  keep  track  of  the  many  different  issues,  activities, 
participants,  and  costs  being  addressed  by  a  large  team,  Boeing 
implemented  a  strong  program  management  system  to  collect  the 
cost  and  technical  progress  data  and  information  from  the 
design/build  groups  and  provide  it  to  the  managers  for 
evaluation.  The  organizational  structure  of  Boeing's  concurrent 
engineering  development  approach  and  the  communication  tools 
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available  produced  an  environment  by  which  the  effort  could  be 
effectively  tracked  and  managed. 

Development  Environment 

Boeing's  concurrent  engineering  approach  helped  foster  an 
environment  conducive  to  appropriate  and  successful  systems 
engineering  practices.  A  prime  feature  has  been  emphasis  on  the 
customer.  Boeing  gave  actual  and  potential  customers  major  voices 
in  the  development  of  the  777.  Airlines  had  early  and  continuous 
involvement  in  the  design  process,  and  it  has  resulted  in 
significant  operational  improvements  that  have  increased  the 
aircraft's  appeal  to  them.  [Aviation  Week,  1994]  Through  initial 
market  surveys,  Boeing  learned  the  airline's  basic  desires.  Many 
of  the  airlines  wanted  a  large  transport  that  uses  an  engine 
whose  design  is  essentially  a  modified  version,  or  derivative,  of 
an  already  existing  engine  in  use,  not  a  totally  new  one.  Boeing 
complied  by  giving  the  airlines  the  choice  of  a  derivative  engine 
from  Pratt  &  Whitney  and  two  new  engines  from  General  Electric 
and  Rolls-Royce. 

It  was  through  full-time,  on-site  airline  advisory  teams  at 
Boeing,  though,  that  the  initial  airline  customers,  United,  All 
Nippons  Airways,  British  Airways,  and  Cathay  Pacific,  had 
influence  over  lower— level  design  details.  Their  representatives 
were  involved  in  reviewing  the  design  and  developing  flight  and 
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maintenance  procedures.  Based  on  airline  advisory  team  inputs, 
Boeing  developed  a  new  technology  cockpit  derived  from  the 
advanced  747-400  cockpit  instead  of  the  767  as  it  was  planning. 
Airlines  also  played  a  pivotal  role  in  determining  the  width  of 
the  fuselage.  An  airline  representative  also  suggested  the 
folding  wingtips  option  as  a  solution  to  airport  gate 
compatibility  problems  resulting  from  the  wide  wingspan.  While 
the  customers  influenced  some  detailed  design  decisions,  the 
fundamental  requirements  remained  stable  throughout  development . 

In  addition  to  steadfast  requirements,  funding  and  workforce 
levels  followed  a  stable,  long-term  plan.  The  Boeing  Company  had 
totally  committed  itself  to  the  four  and  a  half  year  development 
program,  and  that  included  corporate  funding  to  accomplish  it. 
While  Boeing  was  not  at  the  mercy  of  its  777  customers  for 
immediate  operating  funds,  it  did  attempt  to  secure  as  many 
orders  as  possible  as  early  as  possible  in  hopes  of  recouping 
development  costs  as  quickly  as  possible.  In  fact,  the  program 
was  initiated  with  firm  orders  from  United  Airlines  for  34 
aircraft  and  options  for  34  more,  part  of  the  largest  single 
commercial  order  in  Boeing's  history.  [O'Lone,  1990] 


While  the  777  development  team  was  large,  most  key  designers  and 
managers  remained  throughout  the  development.  This  included  not 
only  Boeing  employees,  but  also  the  major  subcontractors. 
Boeing's  development  approach  depended  partly  on  establishing 
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close,  long-term  relationships  with  subcontractors  and  suppliers 
so  as  to  ensure  the  availability  of  dependable  sources  into 
production.  This  relationship  building  was  also  helpful  in 
fostering  cooperation  among  all  team  members.  The  team  spirit 
encouraged  by  Boeing  and  the  creation  of  multi-disciplinary, 
multi-company  teams  promoted  cooperation  among  all  stakeholders. 

Cooperation  among  all  the  program  participants  was  further 
enhanced  by  the  means  of  communication  and  coordination  provided 
by  Boeing.  As  previously  discussed,  CATIA  gave  nearly  everyone  in 
the  design  process  instant  access  to  the  same  configuration  and 
status  information.  In  addition,  subcontractors  and  vendors  had 
real-time  access  to  select  portions  of  the  data  base.  With  all 
teams  and  the  thousands  of  team  members  using  the  equivalent  of 
the  same  set  of  the  latest  drawings  at  any  given  time,  CATIA  has 
been  serving  as  a  communication  tool  that  promotes  the  concurrent 
engineering  approach.  While  the  111  is  not  the  first  aircraft 
development  effort  adopting  a  "paperless"  design  approach,  it  is 
the  first  commercial  aircraft  program  to  adopt  such  a  practice  as 
completely  as  it  has. 

Communication  was  also  enhanced  by  locating  the  entire  Boeing  111 
development  team  in  the  same  general  area  in  and  around  Seattle, 
Washington.  Also  residing  in  the  Boeing  program  offices  were 
full-time,  on-site  representatives  from  the  major  subcontractors 
and  initial  airline  customers. 
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The  advanced  communication  environment  would  have  had  minimal 
impact  unless  the  workers  knew  what  their  responsibilities  were. 
In  Boeing's  design/build  team  structure,  every  worker  had  the 
means  to  clearly  understand  his  or  her  responsibilities  because 
each  team  had  a  written  charter  defining  them.  Furthermore, 

Boeing  defined  the  relationships  between  the  different  levels  of 
teams  in  an  official  document  before  the  start  of  FSED.  This 
document,  a  cross  between  a  program  management  plan  and  a  systems 
engineering  management  plan,  has  had  to  be  updated  to  reflect  the 
continually  changing  list  of  design/build  teams. 

While  Boeing  is  a  large  corporation,  and  the  777  development  team 
consists  of  thousands  of  people,  Boeing  provided  the  design/build 
teams  with  the  authority  and  autonomy  to  carry  out  their 
responsibilities  without  major  interference  from  corporate 
headquarters.  Furthermore,  because  of  the  speed  of  CATIA  and  its 
wide  access,  design  changes  were  processed,  evaluated,  and 
implemented  quickly,  thereby  contributing  to  an  environment 
allowing  a  fairly  high  degree  of  design  flexibility. 

This  level  of  flexibility  and  autonomy  was  also  made  possible  by 
the  expertise  and  experience  of  the  workforce.  Boeing  has  been 
designing  and  producing  passenger  jets  for  over  30  years,  and  the 
technical  knowledge  gained  during  that  time  has  been  resident  in 
the  team  since  the  Ill's  inception.  Furthermore,  Boeing  employed 
many  workers  with  experience  gained  in  the  747—400  development 
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conducted  in  the  mid-  to  late-1980s.  In  addition,  many  of  the  key 
subcontractors  are  specialists  at  developing  particular  items, 
and  they  also  have  long  histories  of  successful  aerospace 
products  lines.  The  777  development  team  can  therefore  be  viewed 
as  being  highly  capable. 

Since  profitability  of  the  111  program  is  dependent  on  the  number 
of  aircraft  produced,  Boeing  would  like  to  sell  as  many  as 
possible.  Future  sales,  however,  are  highly  dependent  on  how  well 
the  initial  aircraft  perform  in  service.  While  Boeing's 
reputation  for  developing  good  passenger  aircraft  helped  it  gain 
initial  customers,  the  company  is  accountable  for  the  performance 
of  the  777.  In  order  to  ensure  that  its  customers  are  provided 
with  the  performance  they  ordered,  Boeing  warranties  key 
performance  parameters  and  workmanship. 

Summary /Conclusion 

The  111  represents  a  new  approach  to  designing  and  building 
aircraft  by  Boeing.  It  was  made  possible  by  a  new  development 
philosophy,  new  computer-based  design  and  information  technology, 
and  a  new  cooperative  attitude  on  the  part  of  the  company 
integrated  with  the  recognized  practices  and  tools  of  aerospace 
design.  The  start  of  flight  testing  on  schedule,  the  high  degree 
of  customer  involvement  throughout  all  phases  of  development,  and 
delivery  of  the  first  production  unit  on— time  suggest  a  design 
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Table  3.1-2  777  Systems  Engineering  Fundamentals  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

REQUIREMENTS  DEVELOPMENT 

0-10 

2 

10 

20 

INCIPIENT  SYSTEM  DESIGN 

0-10 

2 

9 

18 

EVALUATING  ALTERNATIVE 
CONCEPTS  AND  DESIGNS 

0-10 

1 

10 

10 

MAKE-OR-BUY  DECISION 

0-10 

1 

10 

10 

VALIDATION 

0-10 

1 

9 

9 

VERIFICATION  AND  INTEGRATED 
TESTING 

0-10 

1 

10 

10 

CONFIGURATION  MANAGEMENT 

0-10 

1 

10 

10 

MANUFACTURING  CONSIDERATIONS 

0-10 

1 

9 

9 

SYSTEMS  INTEGRATION  AND 
TECHNICAL  MANAGEMENT 

0-10 

1 

10 

10 

LIFE  CYCLE  CONSIDERATIONS 

0-10 

1 

10 

10 

PROGRAM  MANAGEMENT 

0-10 

1 

*  H_ s’** 

10 

10 

SYSTEMS  ENGINEERING 
FUNDAMENTALS  TOTAL 


126 
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Table  3.1-3  777  Development  Environment  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

EMPHASIS  ON  THE  CUSTOMER 

0-10 

1 

10 

10 

STABILITY  OF  REQUIREMENTS 

AND  CONFIGURATION 

0-10 

1 

8 

8 

FUNDING  AND  WORKFORCE -LEVEL 
STABILITY 

0-10 

1 

9 

9 

STRONG  SUPPORT  FOR  PROGRAM 

0-10 

1 

10 

10 

CONTINUITY  OF  CORE 

DEVELOPMENT  TEAM 

O 

1 

H- 1 
O 

1 

9 

9 

STABILITY  OF  ORGANIZATIONAL 
STRUCTURE 

0-10 

1 

9 

9 

COOPERATION  AMONG  ALL 
STAKEHOLDERS 

0-10 

1 

9 

9 

EFFECTIVE  COMMUNICATION 
WITHIN  TEAM 

0-10 

1 

9 

9 

FLEXIBILITY  AND  AUTONOMY 

0-10 

1 

7 

7 

WORKFORCE  EXPERTISE  AND 
EXPERIENCE 

0-10 

1 

8 

8 

ACCOUNTABILITY  FOR  SYSTEM 
PERFORMANCE 

0-10 

1 

8 

8 

- P — !  1 

DEVELOPMENT  ENVIRONMENT  |r  |§| |  |  | 

TOTAL 

sin 

96 
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Table  3.1-4  777  Design  Difficulty  scores. 


Elements 

Range 

Score 

TYPE 

0-15 

9 

KNOWLEDGE  COMPLEXITY 

0-10 

6 

STEPS 

0-10 

9 

QUALITY  IMPLEMENTATION 

EFFORT 

0-10 

9 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

4 

SELLING  PRICE  CONSTRAINT 

0-5 

4 

DESIGN  DIFFICULTY  TOTAL 

0-55 

41 

Table  3.1-5  777  Resources  scores. 


Elements 

Range 

Score 

COST 

0-15 

12 

TIME 

0-10 

7 

INFRASTRUCTURE 

0-10 

8 

RESOURCES  TOTAL 

0-35 

27 
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3.2  CASE  STUDY:  LOCKHEED  F-117  STEALTH  FIGHTER 


The  F-117  Stealth  Fighter  is  a  transonic,  single-pilot  Air  Force 
jet  developed  by  Lockheed  with  the  primary  mission  of  penetrating 
enemy  airspace  at  night  undetected,  destroying  specifically 
designated  high  value  targets,  and  surviving.  [Miller,  1993]  The 
design  of  the  F-117  takes  advantage  of  low  observability 
technology,  rendering  the  aircraft  nearly  invisible  to  radars  and 
other  detection  sensors.  The  F-117  represents  a  revolution  in 
aircraft  design,  and  it  is  the  first  aircraft  in  which  low 
observability,  or  stealthiness,  was  the  main  design  objective. 
[Goff,  ca  1992]  While  the  Stealth  Fighter  employed  breakthrough 
technology,  the  development  risk  was  minimized  by  the  use  of 
existing  subsystems  and  technologies  throughout  the  rest  of  the 
aircraft.  Furthermore,  it  was  developed  in  an  environment 
conducive  to  generating  innovative  designs. 

Development  History,  Design,  and  Performance 

What  eventually  became  the  F-117  program  began  as  a  low-level 
investigation  sponsored  by  the  Defense  Advanced  Research  Projects 
Agency  (DARPA)  into  stealth  technologies  in  1974.  DARPA  had 
requested  that  several  aircraft  manufacturers  conduct  competitive 
preliminary  studies  addressing  a  fighter  with  significantly 
reduced  radar  detectability.  Technologies  to  counter  radar 
detection  and  tracking  had  been  investigated  to  a  limited  degree 
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since  the  mid-1950's.  For  example,  the  reduction  of  radar  cross 
section  was  a  goal  of  the  Lockheed  A-12  and  SR-71  development. 
However,  it  was  not  until  the  Vietnam  War  and  the  threat  of  radar 
guided  surface-to-air  missiles  that  a  compelling  need  was 
recognized  by  military  planners. 

After  the  studies  were  completed,  two  of  the  contractors  were 
invited  to  participate  in  a  competitive  effort  to  develop  and 
test  a  stealth  aircraft.  The  team  from  one  of  the  two 
contractors,  Lockheed,  created  a  revolutionary  stealth  design 
computer  code  and  performed  static  low  observability  model 
demonstrations  supporting  the  results  of  the  code.  Based  on  these 
achievements,  Lockheed  won  the  competition  in  April  1976  to 
design,  develop,  and  test  two  advanced  technology  prototypes  for 
flight  testing  in  an  18  month  classified  advanced  technology 
development  program  called  Have  Blue. 

The  program  called  for  a  variety  of  tests,  including  radar  cross 
section  and  wind  tunnel  model  tests  of  the  prototype  design, 
qualification  and  proof  tests  for  various  systems  and  subsystems, 
preflight  testing  of  the  assembled  aircraft,  and  flight  testing. 
The  first  flight  of  the  unorthodox  looking  Have  Blue  prototype 
was  December  1,  1977,  20  months  after  contract  award.  The  second 
prototype  incorporating  modifications  over  the  first  was 
delivered  seven  months  later,  and  flight  testing  continued  until 


the  summer  of  1978. 
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Based  on  the  results  of  the  prototype  flight  tests,  the  Air  Force 
awarded  a  fixed-price  full-scale  engineering  development  (FSED) 
contract  and  fixed-price  production  contract  under  the  program 
name  SENIOR  TREND  to  Lockheed  in  November  1978  and  December  1979 
respectively  for  concurrent  development  and  production  of  five 
FSED  prototypes  and  15  production  units  based  on  the  HAVE  BLUE 
prototype  design.  The  first  F-117  flight  occurred  on  18  June 
1981,  31  months  after  start  of  FSED,  and  it  became  operational  in 
October  1983,  a  total  of  60  months  from  FSED  start.  (See  Figure 
3.2.)  The  schedule  for  first  flight  and  initial  operational 
capability  by  the  Air  Force  was  about  a  year  late  (about  25 
percent),  due  to  the  crashes  of  two  F-117s  and  specific 
development  problems.  The  Air  Force  eventually  bought  a  total  of 
59  F-117s,  which  were  produced  at  a  rate  of  about  eight  per  year, 
with  the  final  unit  was  delivered  in  July  1990.  The  entire 
program  was  classified  and  conducted  in  secret  until  it  was 
acknowledged  to  the  public  in  November  1988. 

Figure  3.2  F-117  development  schedule 
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The  total  cost  of  the  Stealth  Fighter  development  for  1978-1990 
in  then-year  dollars  was  about  $6.27  billion.  This  represents  $2 
billion  for  development  and  $4.27  billion  for  producing  59 
aircraft.  Over  the  life  of  the  development  program,  FSED  costs 
increased  by  53  percent,  according  to  Air  Force  testimony  to  a 
U.S.  senate  subcommittee.  A  significant  amount  of  this  was  due  to 
inflation  and  to  delays  stemming  from  the  crashes  of  two  F-117s. 
[Lynch,  1992] 

The  revolutionary  nature  of  stealth  technology  requirements 
ensured  that  there  would  be  a  variety  of  developmental 
challenges.  The  major  one  was  to  develop  a  stealthy  but  flyable 
and  maneuverable  external  configuration  in  which  to  integrate  the 
aircraft  subsystems.  The  design  also  had  to  meet  minimum  range 
and  speed  requirements.  In  all,  the  F-117  designers  had  to 
address  seven  types  of  observable  signatures  that  can  be  used  to 
detect  an  aircraft:  radar,  infrared,  visual,  contrails,  engine 
smoke,  acoustic,  and  electromagnetic  radiation.  Having  to  deal 
with  all  these  issues  significantly  increased  aerodynamic  and 
subsystem  integration  difficulties. 

Lockheed's  approach  to  moving  stealth  from  concept  to  reality  was 
to  rely  on  off-the-shelf  hardware  when  possible,  modify  existing 
equipment  where  feasible,  and  invent  new  systems  only  when 
required.  [Miller,  1993]  The  objectives  were  to  minimize  system 
development  risks  and  costs,  enabling  the  design  engineers  to 
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focus  on  the  breakthrough  of  reducing  radar  cross  section  without 
having  to  invent  new  avionics  and  engines.  [Miller,  1993]  To 
accomplish  this,  subsystems  and  components  were  taken  from  a  wide 
variety  of  aircraft  existing  at  the  time.  For  example,  most  of 
the  cockpit  avionics  and  the  engines  were  derived  from  the 
McDonnell  Douglas  F/A-18  fighter,  flight  controls  came  from  the 
General  Dynamics  F-16  fighter,  the  landing  gear  and  ejection  seat 
were  from  the  McDonnell  Douglas  F-15  fighter,  the  inertial 
navigation  system  was  adapted  from  the  Boeing  B-52  bomber,  and 
the  environmental  control  system  components  came  from  the 
Lockheed  C-130  transport.  Other  subsystems  and  components  were 
obtained  from  many  other  airplanes  going  as  far  back  as  the  T-33 
jet  trainer  of  the  early  1950s. 

While  appropriate  subsystems  were  readily  available,  answers  to 
ensure  stealthiness  of  design  were  not.  According  to  Lockheed 
officials,  one  of  the  most  difficult  issues  to  solve  involved  the 
four  small  pitot  tubes  that  extend  from  the  aircraft  nose  to 
gather  air  data.  [Hughes,  1991]  The  presence  of  these  four  small 
items  had  a  significant  impact  on  the  size  of  the  radar 
signature .  It  took  engineers  three  years  to  come  up  with  a  design 
solution  to  preserve  the  F-117's  low  observability  with  the  pitot 
tubes. 

Aircraft  shape,  though,  was  just  one  consideration  of  low 
observability.  The  Lockheed  designers  developed  radar  absorbent 
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material  (RAM)  coatings  used  over  the  primarily  aluminum 
structure  and  used  special  composite  materials  as  weapons  bay  and 
landing  gear  doors,  thereby  reducing  the  aircraft's  reflectivity 
of  radar.  Another  challenge  was  to  hide  the  hot  exhaust  of  the 
engines  from  infrared  and  visual  detection.  This  was  accomplished 
by  developing  a  tailpipe  that  would  flatten  and  cool  the  exhaust 
while  simultaneously  shrouding  radar-reflecting  portions  of  the 
plane.  A  variety  of  other  design  features  were  also  added 
throughout  development  to  enhance  low  observability  performance. 

The  result  of  emphasizing  stealthiness  over  aerodynamics  made  the 
F-117  unstable  in  flight.  This  dictated  that  the  flight  control 
system  had  to  be  a  full-time,  computer  controlled,  fly-by-wire 
command  augmentation  system.  [Miller,  1993]  The  only  technology 
with  adeguate  technical  maturity  at  the  time  was  the  control 
system  used  on  the  then  newly  operational  General  Dynamics  F— 16. 
The  only  significant  change  required  was  to  develop  new  control 
laws,  and  they  were  flight  tested  in  a  specially  modified  T-33 
jet  before  the  first  F-117  flight. 

Despite  the  use  of  mostly  off-the-shelf  avionics,  integrating 
them  was  a  major  challenge  made  more  difficult  by  the  stealth 
requirements  which  mandated  substantial  integration  of 
airframe/avionics  design.  An  example  of  an  important  item  with 
integration  difficulties  was  the  Texas  Instruments  Infrared 
Acquisition  and  Detection  System  (IRADS),  the  laser  targeting 
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subsystem  for  the  F-117's  two  2,000  pound  precision  guided  bombs. 
Making  the  turret  openings  of  this  crucial  subsystem  invisible  to 
radar  was  a  major  problem.  The  difficulty  was  finding  a  material 
for  the  turret  coverings  that  would  allow  the  laser  and  infrared 
emissions  to  penetrate  freely  while  remaining  invisible  to  enemy 
radar.  After  trying  several  high  priced  transparent  materials, 
the  eventual  solution  was  an  innovative  use  of  inexpensive  fine 
wire  mesh  covering  the  opening. 

As  a  military  aircraft,  the  F-117,  in  addition  to  stealth  design 
and  weapons,  reguired  other  special  features  not  seen  on 
commercial  aircraft.  Primary  among  these  were  an  ejection  seat, 
defensive  avionics,  and  aerial  refueling  capability. 

The  five  FSED  aircraft  incorporating  all  the  design  features  were 
involved  in  F-117  flight  testing,  and  they  were  modified 
throughout  development  to  test  design  refinements.  Flight  testing 
was  crucial  to  the  design  process,  especially  considering  the 
limited  simulation  tools  available  to  investigate  stealth  issues. 
The  results  of  flight  testing  of  the  first  five  aircraft 
contributed  to  design  changes  that  would  be  implemented  as  the 
succeeding  production  aircraft  were  being  fabricated.  Some  of  the 
initial  off-the-shelf  subsystems  used  to  reduce  development  risk 
were  found  to  be  inadeguate  in  flight  testing  and  had  to  be 
modified  or  replaced. 
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The  F-117  flight  test  program  encountered  some  significant 
problems  in  determining  the  adequacy  of  design  solutions  and  the 
resolution  of  design  problems.  The  difficulties  stemmed  from  the 
fact  that  six  different  test  pilots  were  flying  three  different 
airplanes  during  initial  phases  of  flight  testing.  What  was 
adequate  performance  for  one  pilot  was  in  some  cases  inadequate 
for  another.  Furthermore,  each  test  aircraft  demonstrated  its 
"own  personality  due  to  equipment  installation  tolerances." 
[Lynch,  1992]  Consequently,  flight  testing  got  into  trouble  when 
Lockheed  tried  to  solve  too  many  problems  at  the  same  time.  In 
one  case,  the  problem  was  solved  by  designating  a  single  aircraft 
and  pilot  to  perform  the  particular  tests.  [Lynch,  1993] 

A  considerable  series  of  ground  testing  of  the  test  aircraft  was 
performed  prior  to  the  first  flight  and  continued  throughout 
FSED .  In  addition  to  wind  tunnel  and  radar  cross  section  testing, 
extensive  component,  subsystem,  and  integration  testing  of 
avionics  was  accomplished.  Also,  the  essential  testing  to  certify 
initial  proficiency  of  the  aircraft,  somewhat  equivalent  to 
certification  by  the  Federal  Aviation  Administration,  was 
completed  by  the  initial  operational  capability  date  of  October 
1983.  However,  flight  testing  continued  on  for  several  years  to 
accomplish  objectives  that  would  not  fit  in  the  Air  Force's 
original  schedule.  [Lynch,  1992] 
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The  number  of  F-117s  planned  for  fabrication  was  small,  rendering 
the  development  of  advanced  manufacturing  and  assembly  processes 
and  procedures  financially  impractical.  Consequently,  much  of  the 
F— 117  fabrication  and  assembly  was  done  by  hand.  The  quality 
appears  to  have  been  very  good,  with  one  significant  exception. 
The  first  production  unit  crashed  because  the  airplane's  roll- 
rate  and  pitch  rate  gyroscopes  had  been  crossed  when  they  were 
installed.  This  was  an  example  of  inadequate  inspection,  ground 
testing,  and  possibly  deficient  attention  to  development  of 
manufacturing  and  assembly  instructions.  However,  it  does  not 
appear  to  have  been  evident  throughout  development. 

The  Air  Force  has  been  satisfied  with  F-117  performance;  the 
aircraft  achieved  the  stealth,  range,  and  speed  requirements  set 
at  the  beginning  of  the  program.  While  the  aircraft  was  developed 
with  limited  performance  objectives,  its  design  has  proved  to  be 
somewhat  robust.  For  example,  in  Desert  Storm,  the  F— 117  was  used 
very  successfully  in  a  manner  significantly  different  than  its 
original  purpose:  performing  a  few  specialized  missions.  In  the 
Middle  East,  the  F-117s  were  flying  one  or  two  operational 
missions  per  night  during  much  of  the  air  campaign.  [Kandebo, 

1992]  While  robust  in  one  sense,  such  prolonged  operations 
resulted  in  increased  maintenance  costs. 

The  Air  Force  and  Lockheed  continued  to  modify  and  upgrade  the  F- 
117  throughout  its  production  and  operational  deployment  to 
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enhance  the  F-117's  capability  and  enable  the  aircraft  to  take  on 
new  roles.  The  original  flight  computer,  for  example,  was 
replaced  in  1984  with  a  repackaged  version  of  the  Space  Shuttle 
computer  in  order  to  upgrade  marginal  capability.  Later,  new 
avionics  were  installed,  and  the  engine  exhaust  system  was 
modified.  Some  new  communications  equipment  gave  the  F-117  all- 
weather  mission  capabilities.  The  changes  have  been  made  to 
enhance  the  F-117  operability,  maintainability  and  to  minimize 
support  costs,  not  to  correct  shortcomings  to  initial 
requirements.  [Hughes,  1991] 

Systems  Engineering  Fundamentals 

The  F-117  development  is  characterized  by  the  successful 
accomplishment  of  many  of  the  fundamental  systems  engineering 
practices.  Requirements  definition  was  one  of  them.  The  written 
requirements  placed  on  contract  at  the  beginning  of  FSED  were  the 
result  of  a  series  of  requirements  and  technology  studies  and  the 
advanced  prototype  Have  Blue  development.  Throughout  this  period, 
the  Air  Force  refined  what  it  operationally  required  based  on 
enemy  capabilities.  Furthermore,  the  program  was  technology 
driven,  and  mostly  critical  performance  and  safety  parameters 
were  specified  as  requirements. 

The  Lockheed  development  team  defined  system  and  subsystem  models 
and  allocated  requirements.  Functional  requirements  were  defined 
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and  work  broken  out  according  to  a  detailed  work  breakdown 
structure.  The  team  also  made  use  of  a  variety  of  design  and 
analysis  models  and  simulations  of  somewhat  limited  capabilities 
in  order  to  evaluate  and  decide  among  alternative  concepts  and 
subsystem  designs.  The  F— 117  was  developed  with  the  design  tools 
available  to  the  aeronautics  community  in  the  mid-  to  late-1970s. 
High  and  low  speed  wind  tunnels,  low  performance  computers  for 
running  simple  simulations  and  analysis  programs,  and 
calculators.  The  design  team  had  no  supercomputers,  and  much  of 
the  initial  design  calculations  were  done  by  hand. 

Despite  their  limitations,  a  variety  of  computer  simulations  and 
physical  models  were  essential  to  development.  One  computer 
program  in  particular,  ECHO  1,  guided  the  stealth  shape  design 
during  Have  Blue  and  the  FSED  program.  ECHO  1  allowed  aircraft 
designers  to  predict  a  radar  return.  This  early  computer  modeling 
tool,  however,  was  limited  to  calculations  in  only  two 
dimensions.  This  meant  that  the  resulting  aircraft  would  have  a 
faceted  design  rather  than  a  smooth,  seamless  one.  [Kandebo] 
Lockheed  tested  their  calculations  with  a  one-third  scale  model 
for  radar  cross  section  studies,  giving  credence  to  computer 
simulation  outputs. 

As  discussed  previously,  prototyping  played  a  major  role  in  F— 117 
development.  The  two  advanced  technology  prototypes  produced 
during  the  Have  Blue  program  were  40  percent  smaller  than  the 
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eventual  production  aircraft  and  simpler.  However,  they  were 
successful  in  their  role  of  testing  certain  performance 
projections  that  had  been  based  on  unvalidated  models. 

Physical  modeling  also  played  a  role  in  production.  Lockheed 
constructed  a  full-scale  wooden  mockup  in  1979  before  assembly 
line  activities  commenced.  This  was  done  so  the  exact  shape  and 
fit  of  each  critical  facet  panel  and  component  could  be  defined. 

Another  indicator  of  successful  systems  engineering  practices  was 
the  definition  of  the  integrated  testing  approach  early  in  the 
development  process.  The  series  of  integrated  avionics  and 
stealth  tests  on  the  ground,  flight  control  testing  on  other 
aircraft,  and  flight  testing  requirements  were  well  documented 
prior  to  start  of  FSED. 

The  Lockheed  team  kept  tight  control  over  the  F-117  drawings  and 
specifications  defining  the  configuration,  including  the  internal 
and  external  interfaces.  A  simple  yet  flexible  drawing  and 
configuration  change  system  was  in  place  that  allowed  fast 
turnaround  of  changes  resulting  from  testing  and  other 
verification  activities. 

To  integrate  the  widely  varied  and  interdependent  work  carried  on 
by  relatively  small  groups  of  Lockheed  employees  and  that  of 
about  500  subcontractors  and  suppliers,  a  strong  systems 
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integration  and  technical  management  presence  was  provided  by  the 
Lockheed  program  manager  and  his  deputies.  They  kept  the  focus  on 
the  reguirements  and  succeeded  in  integrating  the  subsystems  into 
the  stealth  shell. 

While  the  F-117  was  a  complex  system  to  design  and  fabricate,  its 
development  focus  was  somewhat  narrow  and  carried  out  at  the 
expense  of  other  considerations.  With  this  in  mind, 
support ability,  discussed  here  primarily  in  terms  of 
maintainability,  was  considered  in  the  design  process  of  the  F- 
117  but  was  not  a  driving  requirement.  Like  other  performance 
considerations,  it  took  a  back  seat  to  stealth.  Where  stealth 
performance  was  very  good,  critics  contend  the  F-117  is  a  prime 
example  of  what  happens  when  support  issues  receive  low  priority. 
[Aviation  Week,  1990]  Case  in  point:  The  F-117  presents  a 
problem  for  technicians  because  stealth  minimizes  the  number  of 
access  panels  and  openings  in  the  skin  in  order  to  obtain  the 
smallest  radar  cross  section  possible.  The  restricted  access 
makes  the  aircraft  more  difficult,  time  consuming,  and  expensive 
to  work  on.  Maintainability  considerations,  however,  did  have  an 
impact  on  this  matter.  Because  of  feedback  resulting  from 
evaluation  of  the  first  F-117  by  Air  Force  maintenance 
specialists,  more  panels  and  doors  were  added  on  subsequent  units 
to  lessen  the  accessibility  problems.  [Henderson,  1991]  In 
another  recognition  of  supportability  considerations,  Lockheed 
simplified  the  necessary  maintenance  and  reduced  the  frequency 
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with  which  RAM  coatings  would  need  to  be  removed. 

With  the  exception  of  coating  maintenance,  the  F-117  maintenance 
practices  are  similar  to  those  used  for  F-15s  and  F-16s.  This  is 
due  largely  to  the  fact  that  most  of  the  subsystems  came  from 
existing  operational  aircraft.  As  a  consequence,  more  than  95 
percent  of  the  equipment  used  to  support  the  F-117  is  common  to 
that  of  other  Air  Force  aircraft.  [Aviation  Week,  1990] 
Furthermore,  despite  the  unique  requirement  of  coating 
maintenance  and  minimal  access  panels,  the  Air  Force  claims  the 
F-117  maintenance  costs  during  normal  operations  are  comparable 
to  those  of  other  tactical  aircraft.  [Aviation  Week,  1990] 

While  the  emphasis  on  supportability  was  limited,  the  Air  Force 
included  only  limited  supportability  requirements.  Had  the  Air 
Force  levied  strict  requirements,  the  F-117  may  not  look  the  same 
or  have  been  successfully  developed  at  all.  This  was  a  major 
tradeoff  the  Air  Force  customer  was  willing  to  make  to  achieve  a 
revolutionary  breakthrough  in  performance . 

With  regards  to  program  management,  Lockheed  managers  utilized 
sufficient  scheduling,  work,  and  cost  tracking  techniques 
throughout  development  to  assess  technical  progress  and  maintain 


control . 
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Development  Environment 

The  F-117  was  developed  in  an  environment  that  emphasized  the  Air 
Force  customer.  The  F-117  contractor  team  followed  the  golden 
rule:  he  who  has  the  gold,  rules.  Therefore,  Lockheed  was 
responsive  to  Air  Force,  and  kept  its  representatives  informed  of 
all  technical  developments.  Furthermore,  the  F-117  contractor 
team  reviewed  the  design  progress  with  the  Air  Force  customer  at 
a  series  of  major  requirements  and  design  reviews,  and  Air  Force 
pilots,  weapons,  and  maintenance  crews  were  heavily  involved 
during  FSED . 

The  F-117  was  also  developed  in  a  stable,  protected  environment 
that  supported  innovative  solutions.  Due  to  its  revolutionary 
military  nature,  the  effort  had  strong  government  support  at  the 
highest  levels  and  was  carried  out  in  complete  secrecy  without 
micromanagement  from  Congress  or  the  Air  Force.  Additionally, 
funding  from  the  Air  Force  was  stable.  Furthermore,  working 
relationships  between  the  Air  Force  customer  and  the  contractor 
development  personnel  were  non-adversarial ,  and  they  operated 
together  in  a  problem-solving  atmosphere.  [Miller,  1993]  These 
various  factors  were  essential  in  enabling  the  program  to  follow 
very  tight  schedules.  The  classification  of  the  program,  however, 
did  cause  problems  in  hiring  people  since  extensive  and  lengthy 
security  clearance  checks  had  to  be  done  on  anyone  coming  into 
the  program. 
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The  F-117  design  team  was  afforded  the  flexibility  by  the  Air 
Force  to  carry  out  much  of  the  design  process  as  it  saw 
appropriate.  In  addition  to  being  free  from  many  bureaucratic 
considerations,  the  aircraft  was  not  overly  specified,  enabling 
the  development  team  considerable  flexibility  in  achieving  the 
stealth  performance.  In  case  performance  had  not  been  met, 
Lockheed's  contract  with  the  Air  Force  included  warranties 
covering  the  aircraft's  reguired  range,  weapon  delivery  accuracy, 
radar  cross  section,  and  workmanship  defects. 

In  addition  to  the  items  discussed  above,  stability  of  customer 
requirements,  continuity  of  core  team  membership,  ease  of  team 
communication  and  coordination,  and  a  high  level  of  workforce 
expertise  and  experience  were  all  hallmarks  of  the  F-117 
development  environment  at  Lockheed. 

The  F-117  was  developed  in  Lockheed's  Advanced  Development 
Projects  group  called  the  "Skunk  Works."  The  Skunk  Works,  which 
years  earlier  had  developed  the  U-2,  F-104,  and  the  SR-71,  was 
organized  for  the  F— 117  development  as  a  tightly  knit  team  of 
highly  talented,  innovative  and  motivated  engineers  and 
technicians.  These  included  technically  competent  managers  who 
were  able  to  communicate  effectively  with  the  team  and  coordinate 
activities,  enabling  them  to  resolve  design  conflicts  in  a  timely 
manner.  This  environment  was  conducive  to  generating 
technological  breakthrough  designs  and  then  carrying  those 
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designs  through  small-scale  production.  It  is  nearly  a  paragon 
environment  for  fast  prototyping  of  advanced  aircraft  designs. 
Most  of  the  development  environment  characteristics  that  support 
effective  systems  engineering  practices  are  part  of  the  Skunk 
Works  mode  of  operation. 

The  Skunk  Works  approach  is  encapsulated  in  fourteen  points  or 
"rules"  developed  by  its  founder,  Clarence  "Kelly"  Johnson.  These 
principles  can  be  viewed  as  addressing  the  systems  engineering 
philosophy  and  practices  of  the  F-117  development.  Some  of  the 
key  aspects  of  this  streamlined  approach  are  as  follows  [Miller, 
1993] : 

1)  The  program  manager  must  have  complete  control  of  the 
program  in  all  aspects.  It  is  essential  that  the  program 
manager  have  authority  to  make  decisions  quickly  regarding 
technical,  finance,  schedule,  or  operational  matters. 

2)  Strong  but  small  project  offices  must  be  provided  both  by 
the  customer  and  the  contractor.  The  customer  program 
manager  must  have  similar  authority  to  that  of  the 
contractor. 

3)  The  number  of  people  having  any  connection  with  the  project 
must  be  severely  restricted.  This  is  because  more  people 
add  bureaucracy,  and  bureaucracy  makes  unnecessary  work. 

4)  A  simple  drawing  and  drawing  release  system  with  great 
flexibility  for  making  changes  must  be  provided.  This 
permits  early  work  by  manufacturing  organizations,  and 
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schedule  recovery  if  technical  risks  involve  failures. 

5)  The  number  of  required  reports  should  be  minimized,  but 
important  work  must  be  recorded  thoroughly.  Responsible 
management  does  not  require  massive  technical  and 
information  systems. 

6)  There  must  be  a  monthly  cost  review  covering  not  only  what 
has  been  spent  and  committed,  but  also  projected  costs  to 
the  conclusion  of  the  program.  Responsible  management 
operates  within  the  resources  available. 

7 )  The  contractor  must  be  delegated  and  must  assume  more  than 
normal  responsibility  for  obtaining  good  vendor  bids  for 
the  subcontracts  on  the  project.  Commercial  bid  procedures 
are  often  better  than  military  ones.  The  contractor  must 
have  the  essential  freedom  to  use  the  best  talent  available 
but  also  operate  within  the  resources  provided. 

8 )  Responsibility  for  basic  inspection  should  be  given  to 
subcontractors  and  vendors .  The  contractor  should  not  be 
duplicating  inspection. 

9)  The  contractor  must  be  delegated  the  authority  to  test  the 
final  product  in  flight,  especially  in  the  initial  stages 
of  development.  This  is  critical  if  new  technology  and  the 
attendant  risks  are  to  be  rationally  managed. 

10)  The  specification  applying  to  the  hardware  must  be  defined 
and  finalized  in  advance  of  contracting.  Furthermore,  the 
Skunk  Works  practice  of  having  a  specification  section 
stating  clearly  which  military  specification  items  will  not 
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be  complied  with  and  the  reasons  for  not  doing  so  is  highly 
recommended.  Standard  specifications  inhibit  new  technology 
and  innovation  and  are  frequently  obsolete. 

11)  Funding  must  be  timely. 

12)  Mutual  trust  must  exist  between  the  customer  project 
organization  and  the  contractor,  and  there  should  be  close 
cooperation  and  day-to-day  communication.  This  cuts  down 
misunderstanding  and  correspondence  to  a  minimum.  The  goals 
of  the  customer  and  producer  should  be  the  same  -  get  the 
job  done  well. 

13)  Access  by  outsiders  to  the  project  and  its  personnel  must 
be  strictly  controlled  by  appropriate  security  measures. 

14)  Due  to  the  small  size  of  Skunk  Works  development  teams, 
ways  must  be  provided  to  financially  reward  people  based  on 
good  performance  and  not  on  the  number  of  people 
supervised.  Responsible  management  must  be  rewarded,  and 
responsible  management  does  not  permit  the  growth  of 
bureaucracies. 

The  Lockheed  team  followed  these  tenets  during  development. 
Furthermore,  the  fact  that  the  F-117  was  a  classified  program 
facilitated  the  implementation  of  this  systems  engineering 
management  approach . 
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Summary /Conclusion 

The  F-117  can  be  considered  a  successful  design,  and  its 
development  can  be  viewed  as  somewhat  successful  overall. 

Although  the  program  experienced  cost  growth  and  schedule  delays, 
the  stealth  fighter  met  the  primary  performance  requirements 
demanded  by  the  customer  in  the  contract.  The  F-117  design  was 
maximized  for  low  observability  while  still  meeting  minimum  range 
and  speed  requirements.  The  revolutionary  design  was  also  robust 
enough  to  enable  its  use  at  Desert  Storm  in  a  manner  and  at  a 
frequency  not  originally  anticipated.  The  conduct  of  this  program 
in  a  stable,  streamlined  environment  while  strongly  implementing 
the  fundamentals  of  systems  engineering  was  key  to  its  technical 
performance  success. 


Table  3.2-1  F-117  Performance  scores. 


Table  3.2-2  F-117  Systems  Engineering  Fundamentals  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

REQUIREMENTS  DEVELOPMENT 

0-10 

2 

9 

18 

INCIPIENT  SYSTEM  DESIGN 

0-10 

2 

9 

18 

EVALUATING  ALTERNATIVE 
CONCEPTS  AND  DESIGNS 

0-10 

1 

10 

10 

MAKE-OR-BUY  DECISION 

0-10 

1 

10 

10 

VALIDATION 

0-10 

1 

9 

9 

VERIFICATION  AND  INTEGRATED 
TESTING 

0-10 

1 

8 

8 

CONFIGURATION  MANAGEMENT 

0-10 

1 

9 

9 

MANUFACTURING  CONSIDERATIONS 

0-10 

1 

6 

6 

SYSTEMS  INTEGRATION  AND 
TECHNICAL  MANAGEMENT 

0-10 

1 

10 

10 

LIFE  CYCLE  CONSIDERATIONS 

0-10 

1 

6 

6 

PROGRAM  MANAGEMENT 

0-10 

1 

9 

9 

SYSTEMS  ENGINEERING 
FUNDAMENTALS  TOTAL 

I 

'  ■  •. 

l 

‘ 

illlillil 

113 

106 


Table  3.2-3  F-117  Development  Environment  scores. 
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Table  3.2-4  F-117  Design  Difficulty  scores. 


Elements 

Range 

Score 

TYPE 

0-15 

13 

KNOWLEDGE  COMPLEXITY 

0-10 

8 

STEPS 

0-10 

8 

QUALITY  IMPLEMENTATION 

EFFORT 

0-10 

7 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

4 

SELLING  PRICE  CONSTRAINT 

0-5 

1 

DESIGN  DIFFICULTY  TOTAL 

0-55 

41 

Table  3.2-5  F-117  Resources  scores. 


Elements 

Range 

Score 

COST 

0-15 

9 

TIME 

0-10 

7 

INFRASTRUCTURE 

0-10 

8 

RESOURCES  TOTAL 

0-35 

24 
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3.3  CASE  STUDY:  NORTHROP  B-2  STEALTH  BOMBER 


The  Northrop  B-2  is  the  United  States  Air  Force's  newest 
intercontinental  strategic  bomber.  This  highly  complex  aircraft 
using  revolutionary  stealth  technologies  was  designed  to 
penetrate  the  air  defenses  of  the  former  Soviet  Union  and  survive 
in  order  to  deliver  nuclear  weapons.  With  the  end  of  the  Cold 
War,  the  B-2  is  expected  to  be  capable  of  attacking  heavily 
defended  targets  with  precision  guided  conventional  weapons  in 
addition  to  its  primary  nuclear  mission.  The  challenging  and 
costly  integration  of  this  wing-shaped  aircraft  was  made  possible 
not  only  by  breakthroughs  in  stealth  technology,  but  also  by  new 
advanced  manufacturing  processes,  a  powerful  computer  tool  for 
assisting  in  the  design  of  large  hardware  systems,  and  the 
disciplined  application  of  systems  engineering  principles. 

Development  History,  Design,  and  Performance 

The  Air  Force  conducted  concept  development  from  1978  to  1980  of 
what  was  initially  called  the  Advanced  Technology  Bomber  (ATB), 
the  same  time  the  F-117  Stealth  Fighter  was  undergoing  full-scale 
engineering  development  (FSED).  In  1981,  two  contractors, 

Northrop  and  Lockheed,  competed  in  a  source  selection  to  develop 
the  ATB.  While  Lockheed's  proposal  included  a  detailed  point 
design  which  was  almost  at  preliminary  design  review  stage, 
Northrop 's  design  was  much  more  conceptual  and  flexible  with  much 
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less  detail.  [Edward,  1995]  According  to  a  government  engineer 
involved  in  the  source  selection,  the  decision  was  close. 

However,  at  the  conclusion  of  the  competition  in  November  1981, 
the  Air  Force  selected  Northrop,  one  of  the  largest  military 
aircraft  companies  in  the  United  States,  to  further  develop  the 
design  and  take  it  into  FSED. 

Despite  the  radical  new  design  and  manufacturing  techniques 
employed  in  the  aircraft,  the  Air  Force  and  Northrop  followed  the 
higher  risk  overlapping  prototype/production  concept  in  an 
attempt  to  compress  the  time  from  development  to  initial 
operation.  However,  before  fully  embarking  on  FSED,  the  Air  Force 
had  Northrop  go  through  a  pre-FSED  risk  reduction  phase  to 
address  critical  technology  and  producibility  issues.  [Edward, 
1995]  This  phase  ended  in  May  1984  with  the  presentation  of  the 
preliminary  design  review. 

Although  the  B-2  program  started  out  on  a  very  tight  schedule,  it 
was  relaxed  somewhat  after  the  B-l  bomber  program  was  reinstated 
in  1982.  With  a  revised  schedule,  the  first  flight  of  the  planned 
flight  test  program  was  scheduled  for  1987.  However,  it  did  not 
occur  until  July  1989,  about  five  years  after  FSED  start.  The 
first  production  aircraft  was  delivered  to  the  Air  Force  for 
operational  deployment  in  December  1993,  nearly  two  years  late. 
(See  Figure  3.3.)  This  represents  an  FSED  schedule  slip  of  about 
25  percent.  The  delay  was  caused  primarily  by  a  major  redesign  in 


Figure  3.3  B-2  development  schedule 
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1984  and  difficulties  associated  with  developing  numerous  new 
design  and  production  technologies.  [Dornheim,  1988] 

The  government  has  paid  $17.5  billion  in  base  year  1981  dollars 
($24.7  billion  in  then  year  dollars)  for  pre-FSED  and  FSED 
activities  and  the  fabrication  of  six  airplanes,  five  of  which 
will  be  delivered  for  operational  use  after  completion  of 
operational  flight  testing.  Another  $11.8  billion  in  base  year 
1981  dollars  ($19.2  billion  in  then  year  dollars)  is  being  spent 
for  15  more,  the  last  which  will  be  delivered  in  1997.  [CPR, 

1993]  In  addition  to  Air  Force  funding,  Northrop  made  close  to 
$2  billion  in  capital  investment.  The  cost  on  the  cost-plus 
development  contract  has  about  doubled  over  the  original 
estimate.  [Gala,  1994]  However,  much  of  the  cost  growth  is  due 
to  economic  factors  related  to  the  extension  of  the  program  and 
is  not  due  solely  to  unanticipated  problems  and  changes.  [CPR, 
1993] 

When  the  B-2  FSED  contract  was  awarded  to  Northrop,  the  Air  Force 
was  planning  to  purchase  a  total  of  132  development  and 
production  units.  As  a  result,  Northrop  and  its  major 
subcontractors  and  suppliers  structured  its  facilities,  tooling, 
and  personnel  planning  to  support  a  large,  long-term  capital 
intensive  program.  [Scott,  1991c]  In  1990,  however,  the 
Department  of  Defense  reduced  the  planned  production  quantity  to 
75  based  on  the  receding  threat  from  the  Soviet  Union  and  the 
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aircraft's  high  cost.  Then  in  1993,  with  mounting  criticism  of 
the  program's  costs,  Congress  capped  B-2  production  at  20.  (See 
Table  3.3-1.)  However,  based  on  only  20  B-2s,  the  unit  cost, 
which  includes  development  expenditures,  is  projected  to  be  over 
$1  billion  per  aircraft  in  base  year  1981  dollars.  However,  had 
the  original  132  production  quantity  remained,  the  unit  cost  was 
expected  to  be  about  $330  million  in  base  year  1981  dollars  ($525 
million  in  then  year  dollars).  In  1994,  the  last  unit  entered 
production,  and  the  contractors  started  closing  down  parts  of  the 
production  line.  Despite  its  production  cap,  Congress  voted  funds 
for  government  fiscal  year  1995  to  keep  the  B-2  production  line 
viable  for  another  year  while  it  decides  whether  or  not  to 
purchase  more. 


Table  3.3-1  B-2  Production  Quantity  Changes 


Year 

1981 

1990 

1993 

Planned  Production 
Quantity 

132 

75 

20 

The  high  cost  of  the  B-2,  made  significantly  higher  by  government 
imposed  quantity  reductions  and  schedule  stretches,  is  due  to  its 
revolutionary  aircraft  design.  The  bomber  is  essentially  a  flying 
wing  fabricated  out  of  advanced  composites  and  materials  to 
render  its  cross  section  nearly  invisible  to  radar.  The  design  is 
also  meant  to  reduce  infrared,  electromagnetic,  and  acoustic 
detection.  The  only  aircraft  with  a  similar  aerodynamic  shape  was 
the  Northrop  YB-49  test  aircraft  from  the  late  1940s.  The  B-2 
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represents  the  fourth  generation  of  stealth  technology ,  drawing 
upon  the  advances  in  low  observability  design,  materials,  and 
coatings  attained  on  the  SR-71,  the  Advanced  Cruise  Missile,  and 
the  F-117A  Stealth  Fighter  programs,  [Scott,  1990]  Like  those 
earlier  program,  the  B-2's  early  development  occurred  in  complete 
secrecy. 

The  primary  design  challenge  for  the  B-2  program  was  to  attain 
the  desired  degree  of  stealthiness  with  aerodynamics  that  would 
allow  needed  range,  speed  and  payload  to  be  realized.  To  help 
achieve  the  radar  cross  section  requirements,  900  new  materials 
and  processes  were  developed.  [Scott,  1990a]  Included  were  the 
processes  to  make  the  composite  wings,  the  largest  single  piece 
composite  structures  ever  built.  Furthermore,  the  B-2's  shape  was 
generated  with  the  help  of  a  three-dimensional  stealth  analysis 
code  run  on  powerful  computers.  The  resulting  aerodynamic  design 
was  unique  and  untried,  but  24,000  hours  of  wind  tunnel  testing 
and  years  of  simulator  tests  before  the  first  flight  confirmed 
the  aircraft  would  be  stable.  [Scott,  1989a] 

The  other  key  challenge  was  systems  integration.  The  B-2  was 
intended  to  be  flown  and  all  its  numerous,  complex  elements 
operated  by  a  crew  of  two  pilots,  requiring  a  high  degree  of 
integration  and  interaction  among  all  on— board  subsystems.  Within 
the  B-2's  airframe  is  packed  a  wide  array  of  subsystems, 
including  both  off-the-shelf  and  new  technology  items.  In 
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addition  to  four  General  Electric  engines  and  avionics  from  a 
variety  of  companies,  the  B-2  has  a  quadruple  redundant 
electronic  flight  control  system,  a  weapons  delivery  system, 
electronic  warfare  system,  aerial  refueling  capability,  and 
ejection  seats  for  the  crew.  The  B-2  also  incorporates  an 
advanced  radar  that  is  a  significant  technological  step  in 
developing  active  sensors  that  are  compatible  with  low  observable 
aircraft.  [Scott,  1991a]  Provisions  were  made  in  the  B-2  for  a 
third  crewmember  in  case  avionics  automation  did  not  provide 
enough  workload  relief  for  the  two-man  crew. 

The  B-2  development  team  encountered  a  range  of  difficulties.  For 
one,  there  were  a  variety  of  problems  in  developing  a  large 
number  of  new  materials  and  processes.  Also,  crew  ejection  seat 
development  and  integration  were  a  significant  problem,  and  it 
helped  delay  the  initial  flight  test.  Weight  increased  higher 
than  planned,  and  it  impacted  the  projected  payload/range 
performance.  Delays  in  fabrication  of  the  first  aircraft  was 
caused  by  problems  such  as  wiring  bundles  either  not  being 
installed  or  being  installed  incorrectly.  Some  of  these  errors 
were  not  discovered  until  complete  electrical  checkouts  were 
ready  to  begin.  [Aviation  Week,  1988] 

Some  other  troubles  were  stress  discontinuities  in  the  cockpit 
windscreen  and  cracking  of  composite  wing  leading  edge  sections. 
The  most  publicized  event,  however,  has  been  the  B— 2's  failure 
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during  flight  testing  in  1991  to  meet  the  radar  signature 
requirement  for  a  few  particular  cross  sections  at  a  narrow 
frequency  band.  The  B-2  has  hundreds  of  cross  sections  of 
interest  which  are  subjected  to  thousands  of  test  measurements 
covering  the  full,  wide  spectrum  of  radar  frequency.  In  the 
domain  where  the  performance  deficiency  occurred,  the  B-2  is 
already  substantially  better  than  the  F-117.  [Rice,  1991]  The 
Air  Force  considers  the  failure  minor  and  not  impacting 
operational  effectiveness.  [Morrocco,  1994]  Furthermore,  in 
order  to  avert  any  major  design  changes  that  could  have  been 
implemented  to  possibly  correct  the  problem,  the  Air  Force 
relaxed  the  requirement. 

A  major  design  change  did,  however,  take  place  earlier  in  the 
program.  The  initial  B-2  design  proposed  by  Northrop  was  intended 
for  high-altitude  operations  only.  However,  the  Air  Force  had 
concerns  about  future  vulnerability  as  threats  and  missions 
changed,  and  it  therefore  added  a  low-altitude  mission  capability 
requirement  after  source  selection.  This  made  the  B-2  an  all¬ 
altitude  aircraft  and  gave  it  more  future  operational  flexibility 
for  nuclear  or  conventional  weapons  missions.  The  redesign 
resulted  in  a  greater  fatigue  life,  more  aerodynamic  efficiency, 
and  lower  weight.  [Dornheim,  1988] 

The  prospect  of  flying  at  low  altitudes  also  increased  concerns 
about  bird  strikes,  resulting  in  a  change  in  wing  leading  edge 
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shape  and  materials.  In  addition,  the  engine  installation  was 
altered  to  protect  against  bird  ingestion.  While  these  changes 
improved  the  flexibility  of  the  B-2,  the  redesign  in  the  middle 
of  development  was  a  major  contributor  to  the  two-year  slip  in 
program  schedule,  costing  about  $1  billion.  [Dornheim,  1988] 

Much  of  the  B-2  program's  early  resources  were  devoted  to 
developing  manufacturing  technologies  necessary  to  build  a  bomber 
with  the  tight  tolerances  necessary  for  a  large  stealth  aircraft. 
This  necessitated  implementing  a  sophisticated  three-dimensional 
computer  aided  design  and  manufacturing  (CAD/CAM)  program,  as 
well  as  new  tool  designs  and  manufacturing  capabilities.  [Scott, 
1990] 


This  new  CAD/CAM  system  was  the  prime  element  of  Northrop 's 
advanced  development  approach.  It  enabled  the  aircraft  to  be 
defined  almost  entirely  on  computer,  nearly  eliminating  paper 
drawings.  This  system  was  also  part  of  a  streamlined  design— to— 
production  tooling  process  that  was  supposed  to  improve  the  parts 
fit  of  the  first  units.  As  a  consequence,  no  full-size  mockups  or 
advanced  technology  prototypes  were  built  since  the  database 
served  as  the  master  model .  The  B— 2  was  the  first  program  to 
implement  a  computer-based  development  system  to  such  a  large 
extent.  However,  its  introduction  and  development  was  difficult 
as  errors  were  discovered. 
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As  a  result  of  the  problems  in  the  CAD/CAM  system,  there  were 
significantly  more  ill-fitting  skin  panels  during  assembly  of  the 
first  B-2  than  expected.  Subseguent  aircraft,  though,  had  better 
fitting  parts  as  the  design/manufacturing  database  was  refined 
and  updated.  [Scott,  1989b] 

In  addition  to  design  and  manufacturing  process  advances, 

Northrop  also  changed  how  the  production  workers  operated  by 
introducing  the  Integrated  Management  Planning  and  Control  for 
Assembly  (IMPCA)  system  in  1991.  It  was  designed  to  virtually 
eliminate  paper  on  the  manufacturing  floor.  Computer  terminals  at 
each  aircraft  work  center  provide  current  and  accurate  diagrams 
and  instructions  for  all  tasks.  [Scott,  1991b] 

After  the  drastic  reductions  in  production  quantities,  some 
manufacturing  and  cost  efficiency  improvements  supporting  large 
scale  production  were  not  implemented  because  the  low  production 
rates  of  about  1.5  B-2s  per  year,  down  from  the  projected  four 
aircraft  per  month,  did  not  justify  the  cost.  Therefore,  the 
production  B-2s,  like  the  FSED  aircraft,  are  essentially 
"handmade."  [Gala,  1994] 

The  B-2  airframe  has  a  service  life  requirement  of  10,000  flying 
hours,  which  represents  about  20  years  in  typical  service. 
However,  if  the  forty  year  old  B-52  is  any  guide,  the  B-2  will 
probably  be  utilized  as  long  or  longer.  The  Air  Force  recognized 
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this  and  had  the  aircraft  designed  to  survive  twice  the  service 
life.  Furthermore,  to  demonstrate  durability  performance,  a  B-2 
airframe  was  subjected  to  the  equivalent  of  40  years  of  normal 
flying,  of  bending,  twisting,  vibrating,  and  flexing.  Only  minor 
modifications  to  the  design  were  required  as  a  result  of  this 
successful  series  of  tests.  [Scott,  1992b] 

Despite  the  high  cost,  the  Air  Force  operational  user,  the  Air 
Combat  Command,  is  quite  satisfied  with  the  B-2.  The  B-2  has  a 
range  of  about  6,600  nautical  miles  unrefueled  and  over  10,000 
nautical  miles  with  one  in-flight  refueling,  enabling  it  to  reach 
nearly  any  place  on  earth.  The  aircraft  meets  the  original 
mission  requirements,  even  though  a  few  of  the  specification 
requirements  on  contract  did  not.  [Gala,  1994] 

Systems  Engineering  Fundamentals 

A  high  degree  of  planning  went  into  developing  the  B-2.  The 
foundation  of  this  planning  was  a  strong  set  of  requirements  at 
the  beginning  of  the  program.  As  the  customer,  the  Air  Force  was 
responsible  for  defining  what  the  ATB,  later  the  B-2,  was 
supposed  to  be  able  to  do.  The  top  level  requirements  were 
developed  in  response  to  the  Air  Force  Mission  Needs  Statement 
( MNS ) ,  the  formal  document  approved  by  the  Joint  Requirements 
Oversight  Council  (JROC)  and  the  Chief  of  Staff  of  the  Air  Force 
( CSAF ) ,  indicating  an  actual  or  projected  operational  deficiency. 
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Using  this  document  and  the  input  from  the  user  at  the  time,  the 
Strategic  Air  Command,  the  Air  Force  developing  organization 
defined  a  set  of  performance  requirements  in  a  draft  system 
specification  that  was  given  to  the  two  competing  contractors. 
This  system  specification  focused  on  performance  and  stayed  away 
from  functional  requirements,  giving  the  contractors  greater 
flexibility  in  defining  a  design  solution.  The  B-2  program  was 
one  of  the  first  large  Air  Force  programs  driven  primarily  by 
performance  requirements  specifications.  [Edward,  1995]  A 
consequence  of  this  was  a  smaller  number  of  engineering  change 
proposals  during  development  than  normal. 

To  help  develop  detailed  requirements,  Northrop  and  the  Air  Force 
used  a  variety  of  modeling  and  simulation  programs  covering 
threat  assessments,  performance  objectives,  and  affordability. 
[Modeling  and  Simulation  Users  Survey,  1994]  The  detailed 
requirements  were  documented,  but  this  was  done  mostly  by 
Northrop,  not  the  government. 

Since  Northrop  did  not  have  many  details  of  the  F-117  to  use  as  a 
guide,  the  company  used  the  results  of  the  threat  simulations  and 
drew  upon  its  prior  work  in  the  area  of  low  observability, 
primarily  its  three-dimensional  radar  cross  section  code,  to  form 
its  design  concept.  The  eventual  concept  was  broken  down  in  a 
structured  manner  into  a  work  breakdown  structure,  as  is  normally 
required  at  the  beginning  of  an  Air  Force  systems  development 
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effort.  This  decomposition  approach,  which  addressed  both  product 
and  functional  issues,  helped  guide  the  definition  of  interfaces 
and  the  allocation  of  requirements  for  the  incipient  system 
design.  Northrop  provided  its  initial  version  of  this 
decomposition  as  part  of  its  proposal  to  the  government,  and  it 
updated  it  as  necessary  throughout  development.  Furthermore,  the 
company  established  reliability  requirement  allocations  early  in 
the  program.  [Edward,  1995]  These  allocations  were  essential  in 
helping  identify  candidate  alternatives. 

As  part  of  source  selection,  the  Air  Force  evaluated  and  chose 
between  two  design  concepts.  Even  before  then,  however,  Northrop 
had  gone  through  a  process  of  evaluating  alternatives  itself. 
Using  computer  models,  Northrop  was  able  to  assess  a  variety  of 
design  approaches  by  running  a  series  of  iterations  and 
evaluating  the  performance  and  cost  tradeoffs.  The  refined 
concept  is  what  was  proposed  to  the  Air  Force.  In  addition, 
alternative  subsystem  design  approaches  and  parameters  were 
evaluated  using  computer  simulations  and  analyses  throughout  B-2 
development. 

Along  with  performing  design  tradeoffs,  Northrop  also  had  to 
decide  where  the  parts  or  software  items  would  be  produced.  Since 
Northrop  was  primarily  an  experienced  aircraft  integrator,  it  did 
not  possess  all  the  design  and  manufacturing  capabilities 
necessary  to  perform  the  effort  by  itself.  For  example,  it  was 
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not  in  the  business  of  developing  and  building  engines. 

Therefore,  it  was  predisposed  to  purchase  many  items  outside  of 
the  company  and  its  Boeing  and  LTV  partners.  Furthermore,  as  a 
classified  program,  there  were  limitations  as  to  where  components 
and  subsystems  could  be  obtained.  While  the  B-2  make-or-buy 
decision  process  was  constrained,  is  was  carried  out  as  required 
by  the  government,  and  it  resulted  in  a  large  number  of 
subcontractors  and  suppliers. 

When  it  came  to  validating  B-2  requirements,  both  the  Air  Force 
and  Northrop  had  major  roles.  The  Air  Force  followed  its  formal 
procedure  for  validating  requirements,  and  it  culminated  in  the 
approval  by  the  JROC  and  the  CSAF  of  the  Advanced  Technology 
Bomber  MNS  and  later  an  operational  requirements  document  as 
discussed  previously.  Northrop' s  validation  activities  to 
determine  if  a  real  world  solution  existed  centered  around  a 
variety  of  threat  and  low  observables  simulations  that  were 
continually  refined  throughout  development.  An  aircraft  with  the 
survivability  characteristics  of  the  B— 2  had  never  been  built 
before.  The  F-117,  a  good  guidepost,  was  much  smaller  and  with 
less  low  observable  performance  than  that  sought  for  the  B— 2 . 
Whether  or  not  the  B— 2  design  could  be  made  to  perform  at  a  level 
completely  effective  to  the  customer  was  an  issue  that  required 
evolutionary  requirements  definition  and  evaluation  throughout 
requirements  development  and  concept  development  phases. 
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To  help  determine  conformance  of  the  B-2  design  to  performance 
requirements,  Northrop' s  team  and  the  Air  Force  conducted  a  large 
amount  of  testing.  As  part  of  Northrop' s  verification  strategy, 
the  company  developed  sophisticated  laboratories  to  perform 
component  and  subsystem  testing  early  in  the  development  process. 
Nearly  a  million  hours  of  test  time  accumulated  through 
environmental  stress  screening,  ground  avionics  testing,  and 
flights  of  a  C-135  avionics  testbed.  Extensive  wind  tunnel, 
avionics,  flight  controls,  computer  systems,  qualification,  and 
acceptance  testing  greatly  reduced  the  number  of  program 
unknowns,  despite  the  extensive  use  of  new  materials, 
technologies,  and  manufacturing  processes.  [Scott,  1990]  The 
final  result  of  these  efforts  was  early  elimination  of  many 
reliability  problems  that  normally  arise  during  flight  testing 
and  initial  operational  service.  [Scott,  1992a] 

The  B-2s  extensive  four  year  development  flight  test  program 
involved  six  aircraft,  five  of  which  were  planned  for  eventual 
delivery  as  operational  units.  These  B-2s  were  instrumented  for 
testing  during  manufacturing  and  assembly.  Focus  of  the  testing 
ranged  from  envelope  expansion  to  avionics,  comparable  to  a 
commercial  aircraft  flight  test  program.  However,  the  program 
also  included  military  specific  tests  involving  defensive 
avionics,  advanced  radar,  electromagnetic  compatibility,  and  low 
observables.  About  a  quarter  of  flight  testing  was  devoted  to  low 
observables  performance  verification. 
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Northrop  also  planned  from  the  beginning  a  very  extensive  ground 
durability  and  structural  test  series  of  the  B-2  airframe  that 
was  conducted  parallel  to  flight  testing.  Two  complete  B-2 
airframes  without  engines  and  electronics  systems  were  built  for 
the  sole  purpose  of  conducting  these  tests.  The  durability  test 
unit  demonstrated  two  full  service  lifetimes  of  vibration  and 
flexing  with  no  major  structural  damage,  and  the  static  test  unit 
demonstrated  structural  integrity  at  150  percent  of  design 
limits.  [Scott,  1992b] 

With  the  great  complexity  of  the  B-2,  keeping  track  of  and 
controlling  the  interfaces  and  design  configuration  was  of 
critical  importance.  Since  the  B-2  design  was  in  a  three- 
dimensional  digital  database,  both  engineering  and  manufacturing 
worked  with  the  same  drawing  data  base.  Furthermore,  Northrop  had 
formal  procedures  for  controlling  changes.  As  a  result,  Northrop 
performed  effective  configuration  status  accounting  and  control. 
Interface  control,  however,  was  somewhat  more  difficult.  The 
digital  database  did  not  have  the  ability  to  automatically 
identify  interferences  between  components.  However,  it  did  serve 
as  a  forcing  function  to  help  highlight  the  need  for  interface 
control  and  served  as  the  basis  for  those  activities.  [Edward, 
1995] 

As  discussed  previously,  the  early  involvement  by  manufacturing 
in  the  B-2  was  critical  since  a  large  number  of  the  advanced 


126 


features  of  the  design  were  process  dependent.  For  many  of  these 
items,  practical  manufacturing  methods  did  not  exist  at  the 
beginning  of  the  program.  Methods  were  developed  and  were  used  to 
produce  the  development  hardware.  Later  in  development  and 
production,  some  of  the  methods  and  processes  in  the  program  were 
refined  to  reduce  cost.  In  fact,  the  Air  Force  funded 
manufacturing  improvements  to  decrease  machining  time  for  some 
parts  and  produce  others  in  less  expensive  materials.  [Scott, 
1991a] 


In  addition  to  focusing  on  manufacturing  process  development 
early  in  the  effort,  there  was  the  critical  need  to  manage  the 
complex  integration  of  the  subsystems  and  components  into  the 
airframe.  This  was  done  through  a  methodical  approach  with 
sufficient  up  front  planning.  Before  performing  detailed 
integration,  Northrop  first  had  to  complete  the  Air  Force's 
mandated  pre-FSED  risk  reduction  to  address  ten  critical  issues. 
They  included  low  observables  performance,  the  fabrication  and 
use  of  large  composite  sections,  and  engine  inlet  compatibility. 
None  of  the  items  were  show-stoppers,  and  all  were  addressed  to 
the  Air  Force's  satisfaction.  [Edward,  1995] 

FSED  was  then  allowed  to  proceed,  and  systems  integration 
activities  were  conducted  primarily  by  Northrop  with  its 
subcontractor  team  members.  Northrop 's  personnel  were  organized 
along  strict  functional  lines,  but  they  worked  together  to 
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address  interdisciplinary  issues  in  a  loose  structure  centered 
around  zones  of  the  aircraft.  This  zone  management  worked  during 
development  and  mainly  involved  engineering  and  supportability 
personnel,  but  not  non-technical  workers.  The  zone  leader  was 
essentially  the  crew  boss  and  helped  coordinate  the  activities  of 
the  group.  This  interaction  mechanism  was  significantly  enhanced 
by  the  existence  of  the  three-dimensional  CAD/CAM  database.  The 
digital  database  served  as  the  master  model  and  forced  good 
coordination  between  the  members  working  in  a  zone.  Despite  the 
use  of  this  informal  structure  that  evolved  during  the  early 
period  of  the  program,  B-2  development  was  still  driven 
functionally.  Furthermore,  the  people  formally  empowered  to  carry 
out  development  and  integration  activities  were  program  and 
functional  engineering  managers,  not  the  zone  managers.  [Edward, 
1995] 


Northrop' s  development  approach  included  the  conduct  of  design 
reviews,  as  required  by  the  Air  Force.  The  contractor  generated  a 
program  management  plan,  but  it  was  the  Air  Force  that  produced 
the  initial  systems  engineering  management  plan.  Also,  the 
company  did  not  have  a  separate  systems  engineering  organization. 
The  Air  Force  B-2  System  Program  Office  (SPO)  had  one,  and  it 
included  a  team  performing  independent  performance  analyses.  The 
SPO  encouraged  Northrop  to  create  its  own  systems  engineer 
office,  but  it  did  not,  choosing  to  keep  the  activities 
distributed  in  the  functional  organizations.  [Edward,  1995] 
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Life  cycle  considerations  were  taken  very  seriously  during  B-2 
development.  In  fact,  there  was  heavy  supportability  emphasis 
during  requirements  definition,  and  representatives  from  the  Air 
Force's  organization  responsible  for  maintaining  aircraft,  Air 
Force  Logistics  Command,  were  deeply  involved  from  the  very 
beginning.  The  Air  Force  imposed  strict  reliability  and 
maintainability  requirements  on  the  contract,  and  Northrop  was 
required  to  present  its  plan  for  dealing  with  the  issues  in  the 
Reliability  Program  Plan  and  the  Integrated  Logistics  Support 
Plan.  Also,  the  Air  Force  placed  a  high  level  of  emphasis  on 
supportability  testing,  which  primarily  involved  ensuring  that 
the  maintenance  manuals  agree  with  the  aircraft  configuration  and 
actual  procedures.  [Scott,  1991d]  This  approach  gave  logistics 
testing  and  flight  testing  near-equal  priorities.  [Aviation  Week, 
1991] 

Some  outcomes  of  this  focus  on  supportability  were  that  (1)  a 
flight  simulator  was  ready  to  use  before  the  first  flight,  (2) 
the  B-2  was  the  first  new  Air  Force  aircraft  to  enter  service 
with  maintenance  manuals  available  upon  delivery  of  the  first 
unit,  and  (3)  the  projected  maintenance  man-hours  per  flight  hour 
is  expected  to  be  considerably  less  than  the  requirement.  [Scott, 
1992a] 


Keeping  proper  emphasis  on  life  cycle  issues  was  one  job  of 
Northrop 's  program  managers.  To  do  this,  the  company  appears  to 
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have  maintained  a  reasonably  strong  program  management  function 
in  a  classified  environment  to  keep  track  of  progress  on  the 
tremendously  complex  aircraft.  Computer-based  schedules  were 
used,  and  an  effective  cost  reporting  system  was  in  place. 

Program  management  also  held  a  series  of  regular  program  reviews. 
Early  in  the  program,  they  were  quarterly  and  located  at  the 
different  contractor  locations.  Later,  they  became  less  frequent 
and  more  issue  related  as  the  detailed  design  took  shape.  The 
program  managers  did  not,  however,  use  these  meetings  to  usurp 
the  responsibility  of  the  design  managers  to  maintain  oversight 
over  the  details  of  their  technical  areas.  [Edward,  1995]  This 
reflects  the  cooperative  environment  inside  the  secret 
development  world  of  the  B-2. 

Development  Environment 

As  a  large  military  program,  the  B-2  was  developed  with  the  close 
involvement  of  government  representatives.  These  Air  Force  and 
civil  servant  personnel  from  the  B-2  SPO  and  the  Air  Force  Plant 
Representative  Offices  oversaw  all  aspects  of  development,  and 
the  Air  Force's  user  and  maintainer  organizations  interacted 
heavily  with  the  contractor  team  from  the  beginning.  Also,  the  B- 
2  program  represented  a  large  piece  of  Northrop  Corporation's 
income,  and  the  cost  plus  development  contract  included  an  award 
fee  which  incentivized  the  contractor  to  be  responsive  to  the  Air 
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Force.  Therefore,  Northrop  and  its  subcontractors  could  not  help 
but  emphasize  the  Air  Force  customer. 

The  contractors  looked  to  the  Air  Force  primarily  for  performance 
requirements  that  would  not  change.  Early  in  the  B-2  program, 
however,  there  was  a  major  change  with  significant  consequences. 
As  previously  mentioned,  the  Air  Force  added  the  requirement  soon 
after  source  selection  that  the  B-2  have  all  altitude  capability 
to  ensure  future  flexibility,  instead  of  having  only  high 
altitude  capability  as  Northrop  proposed.  Then  in  1983  during 
pre-FSED,  extensive  structural  analysis  by  contractor  and  Air 
Force  engineers  indicated  loads  were  significantly  higher  than 
originally  believed.  This  initiated  a  redesign  to  the  trailing 
edge.  Early  in  1984,  fatigue  and  structural  problems  associated 
with  the  low  altitude  mission  profiles  were  discovered  as  the 
result  of  analysis.  This  resulted  in  moving  the  cockpit  thirty 
inches.  These  design  changes  were  made  before  FSED  and  hardware 
fabrication,  but  they  did  have  significant  cost  and  schedule 
impacts. 

As  in  other  large,  complex  aircraft  programs,  B-2  changes  have 
been  introduced  on  the  production  line,  thereby  creating 
different  production  configurations.  The  B— 2s  are  separated  into 
three  blocks  representing  configuration  sets.  Block  10  and  Block 
20  have  a  total  of  18  B-2s,  and  Block  30  has  two.  The  Block  30 
configuration  reflects  features  and  performance  the  program 
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originally  sought,  and  the  18  other  aircraft  will  be  retrofit 
with  the  capability  at  the  completion  of  unit  20. 

The  majority  of  B-2  development  occurred  while  the  program's 
existence  was  classified.  This  situation  shielded  the  effort  from 
Congressional  and  public  scrutiny  and  contributed  to  an 
environment  of  funding  and  political  stability,  as  well  as 
workforce-level  stability.  This  enabled  the  developers  to 
concentrate  on  designing  the  aircraft  instead  of  constantly 
justifying  the  program.  The  Air  Force  did  not  have  problems 
obtaining  funding  for  the  B-2  until  late  in  development  after  the 
program  was  publicly  acknowledged  in  1988.  While  Congress  was 
very  supportive  when  the  B-2  was  a  classified  program,  political 
pressures  had  changed  some  lawmakers'  positions.  As  the  Cold  War 
ended,  pressures  to  reduce  military  expenditures  prompted  major 
guantity  reductions  of  the  B-2.  Furthermore,  the  rising  unit  cost 
of  a  shrinking  B-2  production  program  have  been  widely  reported 
and  criticized  in  the  media.  In  the  early  1990 's,  the  impact  of 
this  reduced  political  support  was  stretched-out  production  and 
the  gradual  reduction  of  the  workforce— level .  Although  production 
funding  has  been  severely  restricted,  development  funding  was 
never  significantly  impacted. 

Along  with  stable  FSED  funding,  the  B— 2  also  possessed  a  high 
degree  of  continuity  among  key  managers  and  engineers  from  early 
development  to  production.  Contributing  to  this  stable 
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environment,  the  Air  Force  B-2  SPO  maintained  unusual  continuity 
in  its  leadership,  keeping  the  same  military  program  director  in 
place  from  1983  to  1991. 


The  constancy  of  the  core  development  team  mirrors  a  stable 
contractor  organizational  structure  throughout  development.  As 
mentioned  previously,  Northrop' s  leadership  of  the  B-2  contractor 
team  had  a  strong  functional  orientation  which  was  reflective  of 
the  way  the  company  has  been  traditionally  organized.  While  the 
structure  worked  reasonably  well,  the  Air  Force  program  directors 
were  looking  to  change  to  an  integrated  product  development  (IPD) 
approach  which  would  formally  establish  interdisciplinary 
integrated  product  teams  and  force  business  and  technical  issues 
to  be  worked  together.  IPD  was  implemented  in  1991,  when  the 
majority  of  development  was  complete.  Northrop,  however,  has 
resisted  fully  adopting  IPD.  With  the  program  starting  to  close 
down,  the  Air  Force  decided  not  to  enforce  full  compliance. 
[Edward,  1995] 

While  one  aim  of  IPD  is  to  improve  the  level  of  interaction  and 
cooperation  across  functions  and  organizations,  a  spirit  of 
cooperation  was  already  evident  within  the  B— 2  team  early  in 
development,  according  to  two  government  engineers  involved  since 
source  selection.  In  their  opinion,  the  relations  among  the  Air 
Force,  Northrop,  and  the  subcontractors  were  very  good  overall. 
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They  claim  there  was  free  and  open  dialogue,  and  that  the  team 
viewed  everyone  as  true  partners.  They  also  commented  that  the 
contractors  were  not  afraid  to  identify  problems,  and  they  took  a 
proactive  approach  to  propose  solutions.  [Edward,  1995] 

The  degree  of  team  communication  and  coordination,  therefore, 
appeared  to  be  reasonably  high  despite  the  fact  that  the  team  was 
widely  scattered  throughout  the  country.  Furthermore,  very  tight 
security  requirements  made  simple  phone  calls  and  mailing 
packages  difficult,  and  there  was  no  computer  link  between  sites 
initially.  Much  of  the  interaction  came  as  a  result  of  many  on¬ 
site  meetings  throughout  the  country.  In  addition,  the  three- 
dimensional  CAD/CAM  digital  database  was  very  useful,  and  it 
provided  the  basis  of  effective  technical  interaction  at  many  of 
these  meetings. 

Once  the  B-2  became  publicly  acknowledged,  security  procedures 
were  relaxed  that  allowed  old  communication  means  to  be 
reestablished  and  new  ones  to  be  implemented.  An  important  new 
link  established  in  1992  was  the  Logistics  Support  Management 
Information  System,  a  computer  network  connecting  Northrop  with 
subcontractors,  suppliers,  and  the  Air  Force's  supportability 
centers  and  users.  [Scott,  1992a] 

The  widely  scattered  team  members  recognized  what  their 
responsibilities  were.  Northrop  was  the  prime  integrating 
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contractor  with  overall  responsibility  to  the  Air  Force  to  make 
the  system  work.  This  included  the  engine,  which  the  Air  Force 
was  responsible  for  developing  and  buying  directly  from  General 
Electric.  The  various  subcontractors  were  responsible  for  their 
respective  subsystems,  and  they  understood  to  whom  they  reported 
at  Northrop.  The  zone  management  structure  described  earlier, 
though  not  a  formal  organization  with  documented  relationships, 
worked  reasonably  well  due  to  the  cooperation  of  the  team 
members. 

The  ability  of  any  group  to  perform  work  on  the  B-2  was  greatly 
impacted  by  the  security  requirements.  One  of  the  biggest 
problems  in  conducting  the  program  was  getting  people  security 
clearances  in  a  reasonable  amount  of  time,  since  it  could 
sometimes  take  a  year.  However,  once  that  hurdle  was  crossed,  the 
B-2  program  offered  a  high  degree  of  flexibility  and  autonomy  in 
conducting  the  design  work.  This  was  despite  the  fact  that  the 
Air  Force  had  a  SPO  of  several  hundred  people  overseeing  the 
effort.  They  were  not  a  major  hindrance  because  the  Air  Force 
implemented  streamlined  management  techniques.  Instead  of 
imposing  detailed  functional  specifications,  top-level 
performance  specification  were  imposed.  Many  military  standards 
were  either  heavily  tailored  or  else  presented  as  guidelines.  In 
addition,  members  of  the  Air  Force  SPO  worked  well  with  the 
contractors  in  general,  and  they  added  value  by  being  the 
interfaces  in  areas  that  required  government  input.  As  one 
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example,  the  systems  analysts  from  the  B-2  SPO  worked  closely 
with  the  contractors  to  help  identify  and  solve  technical  issues, 
and  they  were  viewed  as  contributing  members  and  not  as  threats. 
[Edward,  1995] 


An  effort  with  the  national  importance,  technical  challenge,  and 
budget  of  the  B-2  warranted  that  the  organizations  involved  place 
their  best  people  on  it.  Northrop  did  this  by  appointing  its  top 
aircraft  designer  to  lead  the  technical  effort.  Further 
motivation  for  Northrop  was  that  as  prime  contractor,  it  had 
total  system  performance  responsibility,  thereby  making  it 
contractually  accountable  to  the  Air  Force.  Furthermore,  as  a 
cost  plus  contract  with  award  fee,  profit  during  development  was 
dependent  on  how  well  Northrop  performed  on  a  yearly  basis. 
Perhaps  most  importantly,  the  Department  of  Defense  required 
extensive  performance  and  workmanship  warranties  be  placed  on 
Northrop  to  enforce  accountability. 

Summary /Conclusion 

The  B-2  is  a  very  complex  and  expensive  system  whose  extended 
development  was  conducted  in  accordance  with  systems  engineering 
fundamentals  in  the  relative  stability  of  the  world  of  classified 
military  programs.  Northrop' s  paperless  development  approach 
using  a  three-dimensional  CAD/CAM  system  contributed  to  that 


136 


stability,  and  it  pioneered,  despite  shortcomings,  the  new 
direction  of  aerospace  development.  This  breakthrough  design 
combines  both  existing  and  advanced  technologies  in  a  highly 
integrated  yet  supportable  stealth  package  with  the  capability  to 
perform  intercontinental  nuclear  and  conventional  weapons 
missions  for  the  Air  Force  well  into  the  twenty-first  century. 


Table  3.3-2  B-2  Performance  scores, 


Figures  of  Merit 

Range 

TECHNICAL  PERFORMANCE  - 
INITIAL 

0-10 

TECHNICAL  PERFORMANCE  - 
MATURE 

0-10 

COST  PERFORMANCE 

0-10 

SCHEDULE  PERFORMANCE 

0-10 

PERFORMANCE  TOTAL 

Score 


8 
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Table  3.3-3  B-2  Systems  Engineering  Fundamentals  scores. 
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Table  3.3-4  B-2  Development  Environment  scores, 
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Table  3.3-5  B-2  Design  Difficulty  scores. 


Elements 

Range 

Score 

TYPE 

0-15 

13 

KNOWLEDGE  COMPLEXITY 

0-10 

8 

STEPS 

0-10 

9 

QUALITY  IMPLEMENTATION 

EFFORT 

0-10 

8 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

5 

SELLING  PRICE  CONSTRAINT 

0-5 

1 

DESIGN  DIFFICULTY  TOTAL 

0-55 

44 

Table  3.3-6  B-2  Resources  scores. 


Elements 

Range 

Score 

COST 

0-15 

12 

TIME 

0-10 

8 

INFRASTRUCTURE 

0-10 

8 

RESOURCES  TOTAL 

0-35 

28 
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Telephone  interview  with  John  Gala,  Deputy  Director  of 
Enginering,  B-2  System  Program  Office,  Wright-Patterson  AFB, 
Summer  1994. 

Telephone  interview  in  February  1995  with  former  and  current 
members  of  the  B-2  System  Program  Office,  Wright-Patterson  AFB, 
Ohio: 

Don  Edwards.  Former  Chief  of  Flight  Systems  Engineering. 

Participated  in  source  selection  in  Fall  1980.  Joined  the 
program  in  late  1981  and  left  in  1987. 

Mark  Wilson.  Chief  of  Structures  until  1993.  Now  Chief  of 
Flight  Systems  Engineering.  Joined  the  program  in  late 
1980  after  contract  award. 

John  Gala.  Deputy  Director  of  Engineering.  Joined  program  in 
1992. 
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3.4  CASE  STUDY:  MCDONNELL  DOUGLAS  C-17  MILITARY  TRANSPORT 

The  McDonnell  Douglas  C-17  is  the  latest  Air  Force  aircraft 
developed  to  meet  airlift  requirements  for  the  Air  Force,  Army 
and  Marine  Corps.  The  C-17  is  intended  to  carry  large  cargo  over 
intercontinental  distances  and  deliver  it  on  short,  unpaved 
runways.  The  aircraft  is  also  expected  to  require  only  minimal 
ground  support,  need  only  a  small  parking  space,  be  able  to 
airdrop  troops  and  equipment,  and  operate  with  a  crew  of  only  two 
pilots  and  a  loadmaster.  The  need  for  such  a  versatile  aircraft 
was  highlighted  by  the  failed  Iran  hostage  rescue  attempt  in  1980 
which  exposed  the  lack  of  sufficient  mobility  to  respond  quickly 
to  emergencies  in  remote  locations.  [McCloud,  1993]  Furthermore, 
the  C-17  is  needed  to  replace  the  30  year  old  C-141  Starlifter 
fleet  that  is  planned  for  retirement  before  the  year  2000. 
Although  most  of  the  subsystems  of  this  original  design  utilize 
proven,  off-the-shelf  technology,  this  complex  aircraft  has  faced 
a  number  of  integration,  performance,  and  management  problems 
during  its  long  development. 

Development  History,  Design,  and  Performance 

The  Air  Force  issued  a  request  for  proposals  (RFP)  to  aircraft 
development  contractors  for  a  new  airlifter  in  late  1980.  In 
August  1981,  Douglas  Aircraft  -  Government  Segment,  a  company  in 
the  McDonnell  Douglas  Corporation,  was  selected  prime  contractor 
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for  the  then  called  C-X  program  in  a  competitive  source 
selection.  However,  contract  award  and  the  start  of  work  were  put 
on  hold  since  the  Air  Force  had  just  been  authorized  by  Congress 
to  build  more  C-5  and  KC-10  cargo  aircraft,  thereby  diminishing 
the  immediacy  of  the  C-17  effort.  A  year  later,  the  Air  Force 
awarded  the  $6.6  billion  fixed-price  full-scale  engineering  and 
development  (FSED)  contract  that  included  options  for  six  initial 
production  units  as  part  of  a  concurrent  development/production 
strategy.  However,  Douglas  was  funded  at  a  low  level,  only  about 
$30  million  a  year,  for  three  years  for  advanced  technology 
development  and  pre-FSED  activities.  Not  until  late  1985  did  the 
government  finally  fund  Douglas  to  begin  FSED.  With  this  delayed 
start  date,  the  Air  Force  expected  production  to  begin  by  1988, 
the  first  flight  to  occur  in  August  1990,  and  initial  operational 
capability  of  12  aircraft  in  early  1992. 

In  January  1988,  the  first  production  contract  option  was 
exercised  according  to  schedule.  However,  the  first  flight  did 
not  occur  until  15  September  1991,  over  a  year  late.  The  first 
production  C-17  was  delivered  to  the  Air  Force  for  operational 
testing  in  September  1992,  and  more  than  fifteen  aircraft  have 
been  delivered  to  the  Air  Force  so  far.  Developmental  flight 
testing  was  completed  in  December  1994,  and  the  C— 17  was  declared 
operational  in  January  1995,  three  years  after  the  originally 
projected  date  and  fourteen  years  after  Douglas  won  the  source 
selection.  (See  Figure  3.4.)  This  represents  an  FSED  schedule 


Figure  3.4  C-17  development  schedule 
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delay  of  about  35  percent  when  compared  to  its  projected  length 
when  it  began  in  1985. 

Along  with  the  stretch  in  schedule  came  an  increase  in  costs.  The 
fixed-price  development  effort  has  an  overrun  of  $571.3  million 
on  $3,499  billion  stated  in  base  year  1981  dollars.  [DAES,  1994] 
This  represents  an  overrun  of  approximately  16  percent.  Since  the 
C-17  was  developed  on  a  fixed-price  contract,  McDonnell  Douglas 
is  liable  for  the  overrun. 

The  unit  price  of  the  C-17,  which  takes  into  consideration  the 
cost  of  development,  depends  to  a  large  degree  on  the  guantity 
the  Air  Force  purchases.  Normally,  the  larger  the  guantity,  the 
lower  the  unit  price.  The  planned  number  of  C-17s  to  be  acquired, 
though,  has  been  drastically  reduced  since  program  start. 
Originally,  the  Air  Force  planned  to  purchase  210  aircraft. 
However,  due  to  delays,  technical  difficulties,  and  perceived 
capability  reductions,  Congress  reduced  the  buy  to  120  in  1990. 
Congress  then  decided  to  limit  production  to  40  units  in  1993. 
(See  Table  3.4-1.)  while  alternatives  to  the  C-17  are 
investigated.  Additional  aircraft  could  be  purchased  if  Douglas 
can  resolve  performance,  cost,  and  schedule  problems  by  the  end 
of  1995.  Based  on  the  purchase  of  120  aircraft,  the  unit  cost  is 
projected  to  be  about  $300  million  (then  year  dollars)  or  $144.5 
million  (base  year  1981  dollars),  which  includes  development 
costs,  but  not  initial  spares,  modifications,  and  outyear 
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operations  and  maintenance  support.  However,  if  production  is 
limited  to  40  aircraft,  the  unit  cost  of  the  C-17  is  estimated  to 
be  $408  million  per  aircraft  (then  year  dollars)  or  $219  million 
(base  year  1981  dollars)  [DAES,  1994] 


Table  3.4-1  C-17  Projected  production  quantity  changes. 


1985 

1990 

1993 

Projected 

Production 

Quantity 

210 

120 

40 

As  the  high  costs  suggest,  the  C-17  has  a  lot  of  capability.  It 
is  a  large  aircraft  with  the  length  of  the  Air  Force's  medium 
airlifter  (C-141)  and  the  fuselage  width  of  the  heavy  airlifter 
( c— 5 ) .  It  is  propelled  by  four  Pratt  &  Whitney  engines  which  are 
essentially  the  same  as  those  used  on  the  Boeing  757  since  1984. 
It  contains  many  of  the  same  features  found  on  large  commercial 
airliners,  such  as  advanced,  highly  integrated  avionics  and  fly¬ 
by-wire  flight  controls.  Being  a  military  aircraft,  it  also 
includes  non-commercial  features,  such  as  air  refueling 
capability  to  increase  range,  defensive  avionics,  the  ability  to 
takeoff  from  and  land  at  small  austere  airfields,  the  capability 
to  backup  and  park  unassisted,  and  a  complex  cargo  handling 
system  that  enables  offloading  by  a  single  person.  The  C— 17  was 
designed  to  carry  palletized  cargo  and  heavy,  oversized  items, 
such  as  a  large  battle  tank  or  three  attack  helicopters,  in  a 
single  load.  In  addition  to  its  cargo  transport  role,  the  C-17 
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must  also  be  able  to  perform  medical  evacuation  and  paratrooper 
airdrop.  In  essence,  the  C-17  represents  the  merging  of 
capabilities  of  the  existing  large  and  small  Air  Force  airlifters 
that  perform  a  wide  range  of  missions. 

Even  though  the  C-17  is  an  expensive  and  very  complex  aircraft, 
its  development  was  originally  conceived  as  a  low-risk  venture, 
in  which  technology  proven  on  other  aircraft  would  be  integrated 
together  in  the  new  airframe  design.  But  integration  of  the 
features  on  a  single  aircraft  of  the  size  and  intended 
versatility  of  the  C-17  turned  out  to  be  a  much  greater 
undertaking  than  Douglas  officials  had  anticipated.  [Smith, 

1993b] 


The  C-17  was  designed  to  be  highly  automated  to  enable  operation 
of  all  flying  and  avionics  system  duties  without  a  third  person. 
This  was  made  possible  by  44  interconnected  computers  controlling 
and  integrating  all  the  aircraft's  subsystems.  Understandably, 
avionics  integration  has  been  the  leading  technical  challenge, 
with  primary  emphasis  on  the  mission  computers.  [Gilmartin,  1990] 
There  are  three  identical  mission  computers  in  the  C-17,  and 
there  had  been  some  difficulty  getting  them  into  synchronous 
operation  due  primarily  to  software  problems.  Because  of  these 
and  other  technical  and  schedule  difficulties,  Honeywell,  the 
subcontractor  in  charge  of  designing  and  developing  the  avionics 
suite,  was  dropped  in  1989,  and  General  Electric  was  brought  on 
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board  with  a  different,  more  complex  design.  [Scott,  1991] 

The  C-17  program  is  replete  with  other  examples  of  design 
challenges  and  difficulties.  One  of  them  concerns  the  wings.  The 
wings  were  not  initially  considered  a  major  design  challenge,  but 
they  became  a  large,  expensive  problem.  The  full-size  structural 
test  vehicle  failed  the  wing  structural  150  percent  load  tests  at 
128  percent  of  test  load  limit  in  October  1992.  The  failure  was 
discovered  to  be  the  result  of  an  inadequate  design  caused  by  ( 1 ) 
a  computational  error  by  the  Douglas  engineers,  (2)  optimistic 
assumptions,  and  (3)  high  and  uneven  distribution  of  the  test 
pads  on  the  wing.  [Lynch,  1993]  The  structural  test  vehicle  with 
a  temporary  modification  to  the  wing  failed  the  test  again  in 
September  1993,  this  time  at  144  percent  of  test  load  limit.  A 
new,  permanent  design  judged  to  be  operationally  safe  and 
technically  sound  by  an  independent  consulting  team  was  developed 
and  successfully  tested  in  January  1994,  and  it  was  incorporated 
into  the  production  line. 

Other  major  design  challenges  have  been  to  keep  weight  growth  to 
a  minimum  and  to  get  the  commercially  derived  engines  to  meet 
fuel  consumption  requirements.  The  current  weight  is  about  three 
percent  above  the  original  allocation,  and  the  Pratt  &  Whitney 
engine  fuel  efficiency  is  2.8  percent  below  original  projections. 
[DAES,  1994]  Both  of  these  deficiencies  have  impacted 
payload/range  performance. 
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In  addition  to  a  considerable  number  of  design  problems, 
difficulties  with  Douglas's  manufacturing  operations  have 
contributed  greatly  to  cost  increases  and  schedule  delays.  In  the 
first  years  of  the  C-17  program,  Douglas's  production  system  was 
very  inefficient.  Inaccurate  or  outdated  engineering  drawings  led 
to  thousands  of  manhours  spent  on  doing  rework  and  repair  out  of 
position  on  the  assembly  line,  thereby  adding  costs  and  delaying 
deliveries.  [Lynch,  1994a]  These  problems  were  compounded  by  a 
variety  of  quality  related  issues,  such  as  faulty  rivet  machines, 
flawed  composite  flight  control  surfaces,  and  out-of-round 
fuselage  cross-sections.  Douglas,  however,  did  not  make 
significant  improvements  to  its  production  operations  until  the 
early  1990s. 

As  a  result  of  the  technical  problems  experienced,  the  C-17  does 
not  meet  all  of  its  original  or  current  performance  requirements 
as  demonstrated  in  extensive  flight  testing  using  production 
aircraft.  Among  them  is  the  payload/range  performance,  whose 
shortfall  can  be  attributed  to  aircraft  weight  increase,  engine 
fuel  efficiency  shortfall,  and  increased  aerodynamic  drag.  This 
requirement  has  gone  through  great  changes  since  the  beginning  of 
the  program.  However,  the  C— 17 's  shortfall  in  this  area  does  not 
reflect  a  high  degree  of  customer  dissatisfaction. 

The  initial  Air  Force  requirements,  as  contained  in  the  request 
for  proposal  in  1980,  called  for  the  new  transport  to  carry  a 
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maximum  of  130,000  lb.  of  cargo  for  an  undefined  unrefueled 
range,  later  suggested  at  2,400  nautical  miles.  The  document  also 
expressed  the  Air  Force's  goal  of  carrying  a  maximum  cargo  load 
of  160,000  lb.  In  its  proposal,  Douglas  claimed  that  its  aircraft 
would  carry  up  to  167,000  lb.  of  cargo  an  unrefueled  range  of 
2,400  nautical  miles.  Douglas  won  the  competition  and  agreed  to 
an  official  increase  in  the  specification  payload  requirement  to 
its  higher  proposed  value  of  167,000  lb. 

As  the  C-17  development  progressed,  it  became  apparent  to  Douglas 
and  the  Air  Force  that  some  of  the  requirements,  including 
payload/range,  would  not  likely  be  met.  This  appears  to  have 
caused  the  Air  Force  to  reevaluate  the  performance  requirements 
on  the  C-17  contract  to  see  if  they  were  appropriate.  The  1989 
review  of  requirements  to  determine  if  the  C-17  was  overspecified 
resulted  in  the  relaxation  of  the  specification  requirement  to 
160,000  lb.  for  the  2,400  nautical  mile  mission.  However,  the 
production  C-17s  tested  in  the  early  1990s  did  not  meet  the 
revised  requirement,  either. 

In  1993,  the  Air  Force  reviewed  the  C-17  requirements  again  in 
light  of  the  end  of  the  Cold  War.  Based  on  reduced  wartime 
airlift  requirements  for  Europe,  it  established  a  new  requirement 
of  157,000  lb.  for  2,400  nautical  miles,  and  established  a  new 
key  requirement  parameter  of  120,000  lb.  for  a  3,200  nautical 
mile  mission  with  a  110,000  lb.  threshold.  These  changes  were 
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part  of  a  broad  contractual  settlement  in  which  Douglas  agreed  to 
drop  monetary  claims  against  the  government,  invest  more  money  in 
the  C-17  program,  and  improve  its  management.  A  summary  of  the 
payload/range  requirements  changes  is  shown  in  Table  3.4-2. 

Other  original  specification  requirements  and  goals  were  relaxed 
as  part  of  the  1989  review,  with  minimal  impact  to  capability. 
These  included  changing  the  launch  response  time  from  five 
minutes  to  15  minutes  (the  same  requirement  on  the  C-141)  as  well 
as  relaxing  about  three  dozen  less-significant  contract 
specifications.  [Morrocco,  1991]  The  Air  Force  operational  users 
reviewed  performance  requirements  again  in  1993  and  changed  many 
to  objectives.  [Morrocco,  1994] 

The  Air  Force  had  the  contractual  authority  to  require  full 
compliance  with  the  performance  requirements.  However,  keeping  to 
the  original  specifications  would  have  forced  a  switch  to  even 
more  powerful  engines,  thereby  eliminating  its  commonality  with 
commercial  airline  engines  and  adding  to  future  maintenance 
costs.  [North,  1993]  Furthermore,  this  modification  would  have 
delayed  the  program  even  further,  thereby  threatening  the 
continuation  of  the  program  and  therefore  the  means  to  fulfill 
projected  airlift  requirements. 
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In  addition  to  issues  surrounding  payload/range  requirements, 
there  have  been  questions  about  reliability  and  maintainability 
performance.  Both  are  areas  with  long-term  implications  for  the 
Air  Force.  The  C-17  was  failing  by  a  substantial  margin  to  meet 
three  system  specification  requirements  for  reliability  in 
developmental  and  initial  operational  flight  testing.  They  were 
Mean  Time  Between  Removal  (MTBR) ,  Mean  Time  Between  Maintenance  - 
Inherent  ( MTBM ( I ) ) ,  and  Mean  Time  Between  Maintenance  - 
Corrective  (MTBM(C)).  Not  meeting  MTBR  could  increase  the 
quantity  of  spares  required,  and  not  meeting  the  system  MTBM 
requirements  could  affect  resources  to  support  the  system.  During 
the  last  half  of  1994,  however,  the  reliability  of  the  C-17 
demonstrated  in  operational  flight  testing  dramatically  improved 
and  was  meeting  or  exceeding  system  specification  growth  curve 
requirements.  [EPMR,  1994]  Meanwhile,  demonstrated 
maintainability  performance,  measured  in  Maintenance  Man-hours 
per  Flying  Hour  (MMH/FH)  and  Mean  Man-hours  to  Repair  ( MMTR ) , 
continued  to  be  better  than  the  growth  curve  requirements,  and 
the  C-17  has  been  meeting  its  Mission  Capable  (MC)  Rate  and 
largely  surpassing  its  Mission  Completion  Success  Probability 
(MCSP)  requirements.  [EPMR,  1994]  Douglas  and  the  Air  Force 
expect  improvements  to  continue,  leading  up  to  the  July  1995 
Reliability,  Maintainability,  and  Availability  Evaluation,  a 
critical  hurdle  which  will  help  determine  if  additional  C-17s 


will  be  built. 
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The  C-17  has  either  demonstrated  or  is  expected  to  meet  most  if 
not  all  the  performance  requirements.  Attaining  them  will  provide 
the  Air  Force  a  significantly  enhanced  airlift  capability  over 
what  it  currently  has.  The  Air  Force  customer  expects  the  C-17 
will  be  adequate  for  its  operational  needs  and  considers  it  the 
most  cost  effective  solution  to  meeting  military  airlift 
requirements.  [Morrocco,  1994b] 

As  described  earlier,  mission  flexibility  was  designed  into  the 
aircraft,  and  it  is  a  key  element  in  the  Air  Force's  criteria  for 
satisfaction  with  the  C-17.  The  design  incorporates  lessons 
learned  from  the  operational  experience  of  the  current  Air  Force 
airlifters,  the  Lockheed  C-141,  C-5,  and  C-130.  This  has  resulted 
in  a  design  with  great  cargo  handling  versatility.  Cargo  handling 
items  that  are  optional  on  other  aircraft  are  standard  equipment 
on  the  C-17.  Although  this  versatility  increased  weight  and 
consequently  decreased  potential  payload/range  capability,  the 
Air  Force  accepted  this  tradeoff  to  ensure  the  mission  flexibil¬ 
ity  was  always  available  to  each  aircraft.  [Dornheim,  1993] 

The  Air  Force  also  expects  the  C-17  to  be  in  operation  a  long 
time.  According  to  the  C-17  Prime  Item  Development  Specification, 
"The  C-17  airframe  service  life  shall  be  30,000  flight  hours. 
Utilization  shall  be  based  on  the  mission  profiles  contained  in 
the  C-17  System  Specification.  The  airframe  shall  be  designed  for 
twice  the  service  life."  It  continues,  "The  C-17  shall  have  a 
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useful  life  of  not  less  than  30  years  under  any  combination  of 
operating  service  and  storage  life,  when  operational  service  life 
has  not  been  exceeded."  As  current  Air  Force  aircraft  in  service 
attest,  particularly  the  B-52,  C-141,  and  C-130,  30  years  is  a 
realistic  length  of  time  to  expect  the  aircraft  to  operate. 
Obtaining  a  C-17  design  to  achieve  that  long  life  and  flexibility 
has  been  a  long  and  difficult  process. 

Systems  Engineering  Fundamentals 

McDonnell  Douglas  is  one  of  the  world's  largest  developers  of 
military  and  commercial  aircraft.  Over  the  past  thirty  years, 
it's  companies  have  built  a  wide  range  of  successful  jets, 
including  the  DC-8,  DC-9,  and  DC-10  passenger  transports,  the  F- 
15  and  F/A-18  fighters,  and  the  KC-10  cargo  transport  and  aerial 
refueling  aircraft.  During  this  time,  McDonnell  Douglas  had 
followed  a  traditional  approach  to  aircraft  development.  This 
method,  supported  by  a  functionally  oriented  organizational 
structure,  was  characterized  by  a  high  priority  placed  on  design 
engineering  activities  with  minimal  influence  of  manufacturing 
design  concerns. 

Since  the  C-17  is  a  military  aircraft,  the  Air  Force  was  closely 
involved  in  most  aspects  of  Douglas'  development  activities.  As 
the  customer,  the  Air  Force  defined  its  requirements  for  what 
would  become  the  C-17  in  a  formal  process  that  normally 
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culminates  in  the  issuance  of  a  Mission  Need  Statement  (MNS)  and 
a  document  defining  operational  requirements.  The  MNS  certifies 
that  an  operational  need  exists  and  defines  it.  The  requirements 
generated  from  the  Air  Force  airlift  study  in  1980  formed  the 
basis  of  the  top  level  mission  requirements  in  a  MNS  for  what  was 
at  that  time  designated  the  C-XX  aircraft.  Operational 
performance  requirements  were  defined  afterwards  by  the  Air 
Force,  and  these  were  used  to  define  system  requirements  in 
conjunction  with  Douglas.  They  were  formally  documented  in  a 
system  specification  and  a  prime  item  development  specification. 
As  previously  discussed,  Douglas  allowed  a  more  stringent 
specification  to  be  imposed  than  what  the  Air  Force  originally 
suggested.  This  was  probably  done  by  Douglas  as  a  way  to  help  win 
the  contract  source  selection.  However,  it  had  long-term  negative 
impacts  for  the  program. 

Douglas  had  modeled  its  C-17  system  concept  using  previous 
transport  and  research  aircraft  as  guides .  The  next  step  was  to 
follow  a  structured  approach  to  decompose  the  elements  of  the 
complex  system  into  a  work  breakdown  structure  (WBS),  as  is 
normally  required  at  the  beginning  of  an  Air  Force  systems 
development  program.  This  approach,  which  addresses  both  product 
and  functional  issues,  helps  define  interfaces  and  allocate 
requirements.  Douglas  provided  its  initial  version  of  this 
decomposition  as  part  of  its  proposal  to  the  government,  and  it 
updated  the  WBS  as  necessary  throughout  development. 
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The  Air  Force  selected  among  several  basic  designs  from  several 
contractors  when  it  held  source  selection.  Furthermore,  during 
the  C-17  concept  development  phase,  Douglas  evaluated  alternative 
concepts  and  designs  to  come  up  with  one  it  believed  would  meet 
the  Air  Force's  requirements.  Douglas  continued  performing 
tradeoff  studies  throughout  the  pre-FSED  effort  and  into 
production,  as  it  has  attempted  to  define,  improve,  or  correct 
the  configuration. 

Along  with  performing  design  tradeoffs,  Douglas  also  had  to 
decide  where  the  parts  or  software  items  would  be  produced.  The 
large  number  of  subcontractors  and  suppliers  on  the  C-17  contract 
indicates  that  the  least  expensive  approach  for  Douglas  was  not 
to  do  everything  in-house.  The  company  has  been  primarily  an 
aircraft  integrator  and  airframe  manufacturer,  and  it  has 
obtained  engines  and  avionics  from  outside  companies.  Douglas's 
make-or-buy  decision  process  on  the  C-17,  a  normal  commercial 
practice  in  large  aerospace  firms,  was  also  required  by  the  Air 
Force  contract. 

When  it  came  to  validating  C-17  requirements,  both  the  Air  Force 
and  Douglas  had  major  roles.  The  Air  Force  followed  its  formal 
procedure  for  validating  requirements,  and  it  culminated  in  the 
approval  by  the  Joint  Requirement  Oversight  Council  (JROC)  and 
the  Chief  of  Staff  United  States  Air  Force  of  the  C-XX  MNS 
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discussed  previously.  On  the  C-17  program,  the  Air  Force  also 
evaluated  the  requirements  two  other  times  to  determine  if  they 
were  still  reflective  of  the  mission.  These  validation  reviews 
occurred  over  a  thirteen  year  development  period,  and  they 
resulted  in  changes  that  reflected  changing  needs.  However,  such 
reviews  probably  would  not  have  occurred  if  the  C-17  had  not 
encountered  technical,  cost,  and  schedule  difficulties. 

As  for  Douglas,  it  performed  its  primary  validation  activities 
during  concept  development  before  source  selection  to  determine 
if  its  concept  design  was  feasible.  Douglas  reviewed  the 
structural  capability  of  the  existing  transporters,  integrated 
aerodynamic  features  it  developed  in  the  late  1970s  to  enable 
short  takeoff  and  landing,  conducted  wind  tunnel  tests  of 
resulting  models,  and  performed  other  analyses  to  help  determine 
if  a  real  world  system  could  be  built. 

Once  the  C-17  requirements  and  design  concept  were  validated, 
Douglas  had  to  verify  that  what  it  was  developing  conformed  to 
requirements.  To  do  this,  Douglas  used  extensive  physical  mockups 
of  the  fuselage  and  cargo  sections  to  demonstrate  equipment 
placement  and  cargo  handling  capability,  two  structural  ground 
test  articles  for  static  and  durabilty  tests,  and  a  cockpit  and 
avionics  bay  mockup  called  the  flight  hardware  simulator  for 
integrated  avionics  testing  on  the  ground.  Also,  prior  to  first 
flight,  each  test  aircraft  underwent  the  On  Ground  Aircraft 
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System  Test  (OGAST).  The  OGAST  is  a  hardware-in-the-loop  test 
which  simulates  aircraft  flight  on  the  ground  and  exercises  the 
aircraft's  system  components  together  instead  of  individually. 
Additionally,  Douglas  produced  and  flew  one  FSED  prototype  flight 
test  unit  before  completing  construction  of  four  FSED  flight  test 
units  that  were  to  be  later  delivered  for  operational  use.  This 
was  all  performed  in  accordance  with  the  Air  Force  required 
formal  test  and  evaluation  master  plan  and  formal  test 
procedures . 

C— 17  flight  testing  was  originally  expected  to  be  run  much  like  a 
fast-paced  commercial  transport  program  in  which  Douglas  would 
take  the  lead.  However,  the  Air  Force  later  decided  to  manage  it. 
Commercial  aircraft  test  programs  typically  require  10-14  months, 
while  Air  Force  test  programs  generally  involve  a  slower  paced 
approach  in  order  to  evaluate  safety  requirements  and  previous 
test  data.  [Smith,  1993c]  Furthermore,  military  aircraft  must 
conduct  additional  tests  to  demonstrate  maneuvers  and 
capabilities  which  are  unknown  in  the  commercial  world.  [Lynch, 
1993]  In  the  case  of  the  C-17,  technical  problems  discovered 
during  flight  testing  also  extended  the  effort.  Consequently,  the 
development  flight  test  program  which  began  in  1991  was  finally 
completed  three  years  later. 

Because  the  C-17  has  been  a  highly  concurrent  development  and 
production  program,  Douglas  delivered  aircraft  to  the  Air  Force 
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while  still  evaluating  and  modifying  the  design.  Therefore,  there 
was  no  stable,  approved  baseline  configuration  to  which  the 
aircraft  were  being  built.  As  a  result,  the  first  twenty-eight 
C-17s  were  of  widely  varying  configurations.  [Smith,  1993a] 
Although  Douglas  had  a  system  for  interface  and  configuration 
status  accounting  and  control  much  like  that  of  other  large 
aerospace  contractors,  it  was  not  effective  in  tracking  so  many 
different  configurations.  For  three  months  in  1994,  the  Air  Force 
C-17  System  Program  Office  (SPO)  and  Douglas  formed  a  special 
action  team  to  define  a  stable  configuration  for  the  Air  Force  to 
declare  operational.  The  team  was  also  chartered  to  develop  a 
disciplined  process  to  track  and  status  the  configuration  of  each 
aircraft.  [PPR,  1994]  The  team  defined  an  operational  baseline 
and  formed  the  Integrated  Configuration  Management  Database  which 
provides  instantaneous  status  information  to  interested  program 
participants . 

Some  of  the  configuration  changes  have  been  prompted  by 
difficulties  experienced  on  the  production  floor.  While  the 
Douglas  manufacturing  organization  conducted  the  C-17  production 
planning,  it  did  not  have  a  large  amount  of  influence  with  the 
design  engineers.  Therefore,  the  components  and  assemblies  of  the 
C-17  were  not  designed  with  ease  of  manufacturability  as  a 
primary  design  driver.  This  is  partly  reflected  in  the  high 
rework  and  repair  rate,  especially  on  the  first  C-17s.  Forty 
percent  of  the  labor  hours  on  the  first  two  C-17s  went  to  repair, 
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rework,  and  nonstandard  work.  [Smith,  1993b]  As  it  overran  the 
development  contract,  Douglas  had  great  financial  incentives  to 
recoup  the  losses  during  production.  Therefore,  Douglas  started 
developing  and  evaluating  parts  redesigns  aimed  at  reducing  the 
cost  of  the  remaining  production  units. 

While  being  an  expensive  aircraft,  the  C-17  does  not  advance  the 
state-of-the-art  in  individual  technology  areas.  It  does, 
however,  involve  the  sophisticated  integration  of  a  lot  of 
technologies  in  a  type  of  aircraft  Douglas  had  not  worked  on 
before.  Unfortunately,  the  company  failed  to  conduct  adequate 
early  risk  assessments  on  some  of  these  systems.  [Morrocco,  1991] 
Douglas  essentially  underestimated  the  extent  and  complexity  of 
the  development  effort,  and  the  result  has  been  constant  design 
changes  to  correct  problems  and  minimize  the  attendant  schedule 
slips. 

Some  of  Douglas's  difficulties  can  be  traced  to  competing 
priorities  for  limited  company  resources.  By  the  time  Congress 
authorized  FSED  funding  in  1985,  McDonnell  Douglas  was  starting 
to  run  other  major  aircraft  development  programs.  In  1987,  the 
company  was  involved  simultaneously  in  developing  the  Navy  T-45 
trainer  and  the  MD-90  and  MD-11  commercial  transports,  as  well  as 
the  C-17.  These  four  concurrent  programs  overburdened  the 
resources  and  talent  of  a  company  that  had  not  produced  a  new 
aircraft  design  in  over  a  decade.  [Morrocco,  1993]  The  C-17 
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program  had  been  drained  of  the  company  talent  and  management 
attention  that  had  originally  been  assigned  to  it  before  Douglas 
had  won  the  source  selection  in  1981.  The  inadequate  allocation 
of  qualified  engineering  and  manufacturing  personnel  contributed 
to  the  design  difficulties  and  led  to  inefficiency  on  the 
manufacturing  floor. 


To  improve  its  manner  of  developing  aircraft,  McDonnell  Douglas 
decided  to  move  its  companies  from  a  traditional,  functionally 
oriented  development  approach  to  a  team-based  approach  that  would 
improve  the  interaction  between  the  engineering,  manufacturing, 
and  support  disciplines.  In  1989,  McDonnell  Douglas  attempted  to 
implement  its  Total  Quality  Management  System  ( TQMS )  corporate¬ 
wide,  including  Douglas.  However,  it  was  poorly  carried  out  and 
disrupted  the  entire  company,  thereby  slowing  C-17  program 
progress  and  failing  to  improve  the  program's  situation. 

In  order  to  help  resolve  the  extensive  C-17  technical  and 
management  problems  that  continued  to  face  the  contractor  in  the 
early  1990s,  the  Pentagon  forced  McDonnell  Douglas  and  the  Air 
Force  to  fully  and  effectively  implement  an  integrated  product 
development  teaming  approach  in  1993  for  the  completion  of  C-17 
development  and  production  of  the  remaining  units.  This  involved 
extending  the  teaming  concept  already  in  place  at  Douglas  to 
physically  locating  interdisciplinary  team  members  together  and 
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formally  including  Air  Force  personnel.  In  response,  Douglas  and 
the  Air  Force  C-17  SPO  established  a  network  of  nine  integrated 
product  teams,  with  each  of  these  teams  in  turn  overseeing 
subordinate  teams.  [Lynch,  1994a]  This  arrangement  was  formally 
documented  in  an  integrated  product  team  plan.  In  addition, 
Douglas  was  required  to  implement  a  paperless  design  system 
through  use  of  an  advanced  computer  aided  design  network,  much 
like  that  implemented  by  Boeing  during  777  development. 

Although  Douglas  had  significant  technical  and  management 
problems,  the  company  did  follow  most  of  the  Air  Force's  required 
sequence  of  design  reviews  and  fulfilled  most  of  its  required 
documentation  submittals. 

Some  of  these  submittals  documented  how  Douglas  was  going  to 
address  a  broad  range  of  life  cycle  considerations,  as  required 
by  the  Air  Force.  Reliability  and  maintainability  requirements 
were  called  out  in  the  system  specification  with  the  objective  of 
ensuring  mission  accomplishment  and  controlling  maintenance 
costs,  and  plans  addressing  these  areas  were  required.  However, 
an  aggressive  reliability  growth  program  was  not  implemented 
until  1994.  [DAES,  1994]  The  Air  Force  also  called  for  ease  of 
item  accessibility  to  reduce  the  time  and  difficulty  of  repairing 
and  replacing  C-17  components,  and  there  was  maximum  use  of 
built-in  test  features  to  reduce  maintenance  and  troubleshooting 
times.  Training  was  given  importance,  and  Douglas  developed  a 
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flight  hardware  simulator  to  train  flight  crews.  The  result  of 
these  efforts  were  incorporated  into  a  life  cycle  cost  model, 
which  was  used  by  Air  Force  planners. 

The  importance  of  life  cycle  considerations  on  the  C— 17  program 
is  illustrated  by  the  decision  to  forgo  additional  efficiency 
improvements  to  the  engine  design  in  order  to  maintain  its 
commonality  with  its  commercial  counterpart.  By  doing  this,  the 
Air  Force  expects  to  save  on  spare  parts  costs  and  possible 
reliability  and  maintainability  cost  increases.  [North,  1993] 

While  supportability  concerns  have  been  addressed  on  the  C--17 
effort,  they  have  not  always  prevailed.  One  design  result  which 
may  have  a  negative  long  term  impact  is  the  proliferation  of 
computer  languages  in  C— 17  software.  Six  different  computer 
languages  are  used  throughout  the  aircraft,  and  many  subsystems 
contain  more  than  one  language.  This  diversity  is  likely  to 
result  in  excessive  software  maintenance  costs  in  the  long  run. 
[Bond,  1992]  This  approach,  however,  was  consciously  chosen  to 
accommodate  shorter  term  schedule  and  cost  considerations.  The 
Air  Force  plans  to  eventially  convert  all  software  to  ADA. 

In  summary,  although  not  all  the  outcomes  have  been  commendable, 
supportability  issues  did  receive  a  high  level  of  attention 
throughout  C-17  development. 
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Douglas  program  management  had  a  difficult  time  keeping  track  of 
the  program  during  most  of  the  C-17's  development.  In  the  past, 
the  output  of  the  Douglas  cost  schedule  and  control  system  often 
lagged  sixty  days  behind  factory  floor  work.  [Lynch,  1994b]  An 
independent  Department  of  Defense  panel  concluded  as  late  as  1993 
that  Douglas's  business  systems  were  struggling  to  provide  the 
management  visibility  and  control  needed  to  properly  support  the 
program.  [Lynch,  1994a]  Managers  therefore  did  not  have  the 
timely  management  data  to  help  identify  problems.  Furthermore, 
the  program  did  not  have  the  Douglas  management  focus  and 
resources  during  much  of  the  program  to  quickly  resolve  the 
problems  once  discovered.  With  the  implementation  of  the 
integrated  product  teaming  approach  in  1993  came  an  improved 
ability  to  track  and  manage  the  program  for  both  Douglas  and  the 
Air  Force.  This  included  a  new,  computer  based  cost  and  schedule 
accounting  system  in  which  the  reports  are  generated  and  provided 
to  Douglas  management  and  the  government  within  seven  days  after 
the  close  of  the  monthly  accounting  period.  A  key  feature  of  the 
new  C-17  management  operations  is  that  there  is  only  one  set  of 
plans,  teams,  and  schedules,  and  therefore  no  longer  separate 
government  and  contractor  versions .  The  improvement  in  program 
management  effectiveness  in  the  mid-1990s,  due  in  part  to  the 
management  changes  implemented,  also  followed  attendant 
improvements  in  the  development  environment. 
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Development  Environment 

As  is  evident  from  the  discussions  up  until  now,  the  C-17  program 
did  not  progress  smoothly.  Mirroring  the  discord,  and  in  some 
cases  contributing  to  the  problems,  has  been  a  difficult 
development  environment  for  both  the  Douglas  and  the  Air  Force. 

The  C-17  was  conducted  in  the  manner  of  a  typical  Air  Force 
system  development  program.  That  is,  the  Air  Force  customer  was 
involved  in  overseeing  all  aspects  of  the  effort  through 
representatives  in  the  SPO  and  on-site  presence  of  the  Air  Force 
Plant  Representative  Office  (AFPRO).  As  was  typical  at  the  time, 
the  Air  Force  also  specified  in  great  detail  what  it  expected  the 
system  to  do.  Furthermore,  the  Air  Force  expects  to  be  kept 
informed  of  development  progress.  Therefore,  Douglas  could  not 
escape  emphasizing  the  customer.  This  arrangement,  in  place 
during  all  of  development,  allowed  much  input  by  the  Air  Force. 
However,  Douglas  would  not  do  everything  the  Air  Force  wanted 
done  to  correct  problems  due  to  the  overrun  on  the  fixed-price 
contract. 

While  the  Air  Force  was  obliged  to  provide  guidance  on 
requirements,  Douglas  was  fully  responsible  for  design  and 
fabrication.  The  Air  Force  specification  requirements  had 
remained  stable  throughout  most  of  development  until  1989  when 
the  Air  Force  relaxed  some  of  them.  Up  until  that  time,  though, 
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Douglas  constantly  modified  the  detail  requirements  and 
configuration  throughout  FSED  and  production  in  response  to 
problems  encountered  as  the  program  produced  hardware  and 
software. 

A  design  change  that  had  a  major  impact  on  the  program  was  the 
addition  of  a  fly-by- wire  electronic  flight  control  system. 
Douglas  officials,  with  Air  Force  concurrence,  decided  in  1987, 
two  years  after  the  start  of  FSED,  that  the  aircraft  needed  the 
electronic  system  to  realize  its  full  capability  and  meet 
operational  requirements.  [Scott,  1991]  The  change  of  this 
magnitude  in  the  middle  of  FSED,  with  its  attendant  systems 
impacts,  contributed  significantly  to  the  cost  overrun  and 
schedule  delays. 

Another  change  with  significant  consequences  was  the  use  of  more 
powerful  versions  of  the  engine  than  used  originally.  This 
upgrade  was  implemented  late  in  development  to  improve  payload 
capability  due  to  aircraft  weight  growth.  Thrust  for  each  of  the 
four  engines  increased  to  41,700  lb.  from  the  previous  rating  of 
37,000  lb.  The  resulting  increase  in  exhaust  temperature 
unexpectedly  led  to  the  need  for  a  higher-temperature  and  heavier 
material  for  the  wing's  externally  blown  flaps,  since  the  old 
material  could  not  survive  the  new  environment.  This  added  weight 


and  increased  cost. 


169 


Due  to  the  highly  integrated  nature  of  the  C-17,  Douglas  has  had 
to  deal  with  the  rippling  effect  of  other  design  modifications 
and  alterations.  Even  in  1994  production,  the  program  continues 
to  be  adversely  affected  by  issues  such  as  design  and  materials 
changes.  [PPR,  1994] 

Stability  has  also  largely  eluded  C-17  funding  and  workforce- 
levels.  The  start  of  FSED  was  delayed  for  nearly  four  years, 
preventing  Douglas  from  ramping  up  manpower  in  preparation.  When 
FSED  funding  finally  came  through  in  late  1985,  Douglas  had  to 
hire  a  large  number  of  people  in  a  short  period  of  time.  After 
this,  the  workforce-level  was  reasonably  stable  as  Congress 
supplied  funding  during  the  first  three  years  of  FSED  in 
accordance  with  the  long  term  budget  plan.  In  1989,  prompted  by 
slippage  in  the  program  schedule,  cost  increases,  and  performance 
problems,  Congress  started  to  cut  the  program's  yearly  budget  and 
alter  the  production  profile.  As  a  result,  Douglas  started 
experiencing  constant  and  increasing  labor  turnover.  This  had  the 
effect  of  severely  reducing  FSED  production  personnel  in  1992  and 
1993,  then  requiring  a  sudden  ramping  up  again  in  1994. 

In  addition  to  cutting  funds  and  reducing  production  quantities 
throughout  the  early  1990s,  Congress  had  threatened  to  cancel  the 
program  altogether.  The  program  was  placed  on  probation  at  the 
end  of  1993,  as  part  of  a  settlement  between  the  Department  of 
Defense  and  McDonnell  Douglas.  In  this  settlement,  the  contractor 
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was  given  two  years  to  prove  it  can  meet  revised  schedule,  cost, 
and  specification  requirements  and  successfully  complete  flight 
testing.  McDonnell  Douglas  also  agreed  to  spend  an  additional 
$456  million  on  facilities  and  testing,  and  drop  $1.7  billion  in 
claims  against  the  government.  If  technical,  schedule,  and  cost 
performance  has  improved  significanlty  by  the  end  of  the 
probation  period,  the  Air  Force  will  be  allowed  to  buy  more  than 
the  40  C-17s  already  delivered  or  on  order.  [Lynch,  1994a]  If 
not,  Congress  will  fund  off-the-shelf  alternatives,  such  as 
buying  more  of  the  large  C-5s  or  a  military  version  of  the  747 
cargo  freighter.  Needless  to  say,  much  of  the  C-17  development 
and  production  has  been  conducted  in  a  volatile  political 
environment  with  widely  varying  degrees  of  support. 

Douglas  had  problems  keeping  a  core  C-17  development  team 
together  even  before  the  political  turmoil.  The  four-year  delay 
of  FSED  in  the  early  1980s  prompted  some  of  the  key  C-17 
technical  and  management  personnel  involved  with  the  earlier 
design  work  to  find  jobs  in  other  McDonnell  Douglas  programs  that 
needed  the  expertise.  Stability  was  also  impacted  during 
production  in  the  early  1990s  by  constant  labor  turnover  due  to 
union  seniority  rules  which  gave  workers  laid  off  from  Douglas' 
commercial  production  programs  the  right  to  claim  jobs  on  the  C- 
17  line.  [Lynch,  1993]  Furthermore,  in  1989,  implementation  of 
the  TQMS  displaced  most  workers,  and  some  of  the  key  C-17 
personnel  did  not  return.  For  example,  at  the  executive  level, 
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194  positions  were  filled  in  the  restructured  organization  chart, 
but  a  year  later,  fewer  than  50  of  those  executives  remained  in 
the  same  positions.  [Smith,  1993b] 

The  Douglas  top-to-bottom  TQMS  transformation  was  imposed 
suddenly  and  without  advance  notice.  The  company's  organizational 
structure  composed  of  functional  groupings  was  virtually 
eliminated,  and  the  process  of  defining  and  filling  the  new 
management  positions  took  up  to  six  months  to  complete.  According 
to  some  employees,  once  the  positions  were  filled,  many  managers 
were  frequently  moved  from  one  position  to  another.  Furthermore, 
some  did  not  have  the  necessary  technical  qualifications  for  the 
job.  [Smith,  1993b]  Unfortunately,  this  transformation  did  not 
achieve  its  intended  effect,  and  the  C-17  program  continued  to 
founder. 

In  1993,  in  response  to  continued  technical  and  management 
problems,  the  Department  of  Defense  directed  the  C-17  SPO  and 
Douglas  to  implement  a  new  master  plan  for  the  C-17  program  under 
the  auspices  of  an  Integrated  Master  Plan.  This  action 
established  integrated  product  teams  as  part  of  integrated 
product  development  (IPD).  [SAR,  1993] 

If  IPD  had  been  adopted  earlier  in  the  program  it  could  have 
enabled  the  Douglas  engineering  and  manufacturing  workers  to  work 
better  together  than  they  were  used  to  under  the  traditional 
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functional  organization.  Douglas's  relationships  with 
subcontractors  and  suppliers,  while  businesslike,  could  also  have 
been  enhanced  through  product  teams.  As  difficulties  befell  the 
program,  Douglas's  interactions  with  its  subcontractors  were 
negatively  affected.  This  was  especially  true  of  Honeywell,  the 
avionics  developer  and  integrator,  which  was  replaced  in  the 
middle  of  development. 

The  Air  Force  SPO  interacted  heavily  with  Douglas,  as  was 
required  of  its  oversight  role.  Traditionally,  government  program 
representatives  and  contractors  have  had  a  mildly  adversarial 
relationship.  In  the  case  of  the  C-17,  a  strongly  negative 
atmosphere  pervaded  as  the  many  problems  became  apparent  and  both 
sides  blamed  each  other  for  them,  creating  gridlock  and  seriously 
impeding  progress.  [Lynch,  1994a]  The  agreement  to  relax 
performance  requirements  and  avert  Douglas  legal  action  mentioned 
previously,  in  addition  to  the  appointment  of  a  new  Douglas 
program  manager  who  reports  directly  to  the  Chairman  of  McDonnell 
Douglas,  were  intended  to  diffuse  the  poisonous  relations  and 
allow  the  program  to  continue. 

Throughout  development  and  into  production,  the  C-17  program 
members  depended  on  the  standard  means  of  communication,  such  as 
telephones,  mail,  meetings,  and  later  faxes.  Most  of  the  Douglas 
workers  were  based  in  Long  Beach,  CA. ,  and  they  were  located  in 
buildings  within  several  miles  of  each  other,  usually  with  other 
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members  of  their  functional  specialty.  Most  of  the  company's 
major  subcontractors  and  suppliers  have  been  located  across  the 
country,  as  well  as  its  Air  Force  customers.  Douglas  has 
conducted  technical  working  meetings  as  well  as  program  reviews 
and  design  reviews  at  a  variety  of  locations  to  provide  the  team 
members  the  opportunity  to  work  together  to  develop  the  aircraft. 
It  has  also  maintained  representatives  at  subcontractor  plants  to 
enhance  communication,  and  some  of  the  subcontractors  have 
representatives  at  Douglas  for  the  same  purpose. 

As  mentioned  earlier,  Douglas  and  the  Air  Force  implemented  an 
advanced  computer  aided  design  and  manufacturing  system  at  the 
tail  end  of  FSED  in  1994.  As  an  enhancement  to  team 
communication,  it  allowed  much  quicker  transfer  of  the  design  to 
program  participants  and  enhancing  the  speed  and  coordination  of 
configuration  updates. 

Due  to  the  large  presence  of  Air  Force  representatives  and 
extensive  reporting  requirements,  Douglas's  flexibility  and 
autonomy  for  conducting  the  C-17  effort  was  less  than  what  would 
normally  be  available  on  a  commercial  transport  effort.  For 
example,  a  former  Douglas  official  claimed  the  management  control 
system  imposed  by  the  Air  Force  required  excessively  detailed 
expenditure  breakdowns  and  was  expensive  to  implement . 
Furthermore,  it  focused  excessively  on  program  detail  and  led 
some  managers  to  lose  sight  of  the  big  picture.  [Smith,  1993b] 
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Also,  the  Air  Force  decided  to  run  the  test  program  instead  of 
having  Douglas  do  it  as  originally  planned.  This  slowed  flight 
test  progress  down,  especially  when  a  three  flight  per  week 
limitation  was  imposed.  Finally,  the  large  amount  of  effort 
required  to  respond  to  Air  Force  and  Congressional  inquiries 
diverted  the  attention  of  members  of  the  technical  and  management 
team  from  their  primary  work. 

Most  of  Douglas's  workers  initially  assigned  to  the  C-17  effort 
were  technically  qualified  for  their  work.  However,  as  discussed 
earlier,  many  engineers,  managers,  and  production  floor  workers 
with  valuable  technical  expertise  left  the  C-17  program  to  take 
jobs  with  other  companies  during  the  four-year  lag  between  being 
selected  as  prime  contractor  and  full  development  funding.  Then, 
when  FSED  funding  was  authorized  in  December  1985,  a  sharp  surge 
in  commercial  aircraft  orders,  as  well  as  ongoing  military 
programs,  made  it  difficult  for  Douglas  to  find  employees  with 
required  skill  levels.  [Smith,  1993b]  The  presence  of  a  large 
number  of  inexperienced  workers  contributed  to  the  production 
floor  inefficiencies. 

Douglas  is  responsible  for  cost  and  schedule  performance  as  well 
as  the  output  of  its  workers.  Furthermore,  it  is  responsible  for 
the  performance  of  the  C-17s  it  delivers  to  the  Air  Force.  To 
enforce  this  accountability,  large  Department  of  Defense 
contracts  are  required  by  law  to  contain  warranties  providing  the 
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government  recourse  if  the  delivered  item  is  inadequate.  The  C-17 
warranty  is  very  extensive  and  includes  provisions  to  ensure  that 
the  C-17s  delivered  (1)  are  free  from  material  and  workmanship 
defects  at  delivery,  (2)  shall  meet  or  exceed  the  system  level 
specification  reliability,  maintainability,  and  availability 
requirements,  (3)  shall  have  had  all  subsystems,  accessories, 
equipment,  and  parts  installed  according  to  specification,  and 
(4)  shall  have  airframes  that  do  not  develop  structural  defects. 
If  an  item  fails  to  meet  any  of  the  above,  the  contractor  must 
correct  the  failure  at  no  additional  cost  to  the  government. 
[Warranty  Briefing,  ca  1994]  Given  the  difficulties  experienced 
on  the  C-17  program,  such  assurances  are  important  to  the  Air 
Force  in  establishing  confidence  in  the  future  viability  of  the 
aircraft. 

Summary /Conclusion 

As  a  large,  Air  Force-sponsored  systems  development  program,  the 
basic  systems  engineering  fundamentals  were  attempted  to  one 
degree  or  another  during  the  development  of  the  C-17.  However,  as 
previously  indicated,  some  of  the  activities  were  not  well 
performed.  In  addition,  the  environment  in  which  the  systems 
development  was  carried  out  primarily  by  Douglas  was  clearly 
turbulent  in  many  areas,  and  it  significantly  contributed  to  the 
program's  shortcomings.  The  end  results  of  the  C— 17 's  technical 
and  management  problems  were  a  significant  schedule  delay  and  a 
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significant  cost  overrun,  but  only  minimal  performance 
degradation.  In  spite  of  the  difficult  development,  the  C-17 
design  is  a  very  capable  aircraft  and  its  operational  performance 
is  satisfactory  to  the  customer. 


Table  3.4-3  C-17  Performance  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

TECHNICAL  PERFORMANCE  - 
INITIAL 

0-10 

1 

3 

3 

TECHNICAL  PERFORMANCE  - 
MATURE 

0-10 

1 

9 

9 

COST  PERFORMANCE 

0-10 

1 

6 

6 

SCHEDULE  PERFORMANCE 

0-10 

1 

5 

5 

PERFORMANCE  TOTAL 


23 


177 


Table  3.4-4  C-17  Systems  Engineering  Fundamentals  scores. 
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Table  3.4-4  C-17  Development  Environment  scores. 
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Table  3.4-6  C-17  Design  Difficulty  scores. 


Elements 

Range 

Score 

TYPE 

0-15 

9 

KNOWLEDGE  COMPLEXITY 

0-10 

6 

STEPS 

0-10 

9 

QUALITY  IMPLEMENTATION 

EFFORT 

0-10 

6 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

4 

SELLING  PRICE  CONSTRAINT 

'  0-5 

2 

DESIGN  DIFFICULTY  TOTAL 

0-55 

36 

Table  3.4-7  C-17  Resources  scores. 


Elements 

Range 

Score 

COST 

0-15 

10 

TIME 

0-10 

8 

INFRASTRUCTURE 

0-10 

8 

RESOURCES  TOTAL 

0-35 

26 
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3.5  CASE  STUDY:  LEARJET  MODEL  60  BUSINESS  JET 


The  Model  60  from  Learjet,  Inc.,  is  a  mid-sized,  twin-engine, 
transonic  business  jet  designed  for  transporting  up  to  eight 
passengers  and  luggage  on  flights  of  up  to  2,750  nautical  miles. 
Learjet  developed  the  Model  60  in  response  to  findings  from  its 
marketing  research  that  customers  existed  for  a  larger  and  longer 
range  version  of  the  successful  Model  55.  Although  the  company 
failed  to  recognize  the  proper  requirements  initially,  the  design 
group  eventually  developed  an  aircraft  that  met  customer  desires 
by  carrying  out,  often  informally,  many  of  the  fundamentals  of 
systems  engineering. 

Development  History,  Design,  and  Performance 

Model  60  development  began  in  July  1990.  In  just  a  few  months, 
the  program  moved  from  concept  definition  to  full-scale 
engineering  development  (FSED).  The  first  flight  of  the 
engineering  prototype  took  place  11  months  later.  Development 
ended  in  January  1993  with  the  first  delivery  of  a  production 
unit  to  a  customer.  (See  Figure  3.5.)  This  two  and  a  half  year 
development  program  that  included  four  aircraft  was  six  months 
behind  original  schedule  (20  percent)  and  cost  about  $90  million, 
which  is  double  its  initial  estimate  for  the  smaller  scope 
program.  The  delay  and  cost  growth  were  due  primarily  to  the 
requirements  changes  generated  by  the  marketing  department . 
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However,  when  considering  the  cost  estimate  update  after  the  new 
requirements  were  imposed  early  in  FSED,  the  overrun  was  about  20 
percent.  [Etherington,  1994] 

Figure  3.5  Model  60  development  schedule. 

1989  1990  1991  1992  1993 

Reqmts  &  Concept  Dev. 

FSED 
1st  Flight 

Flight  Testing  (Certification) 

1st  Production  Delivery 

Learjet  has  been  designing  and  building  business  jets  for  about 
30  years.  The  company  usually  designs  and  builds  the  airframe  and 
integrates  engines  and  avionics  developed  and  built  by  other 
companies.  The  Model  60,  which  is  the  largest  and  longest  range 
aircraft  Learjet  builds,  is  a  direct  derivative  of  the  Model  55 
that  was  introduced  in  1981.  The  Model  55  underwent  several 
performance  upgrades  throughout  1983  and  1984.  Then  the  Model  55B 
was  introduced  in  1985  incorporating  a  digital  flight  deck  and 
increased  takeoff  weight.  The  Model  55C  was  introduced  later,  and 
it  incorporated  the  earlier  aerodynamic  and  performance 
improvements  as  well  as  a  few  others.  The  Model  60  retains  the 
Model  55  series  enhancements,  but  adds  new  elements  to  the  same 
basic  airframe.  [North,  1993]  The  Model  60  development  is 
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classified  as  a  redesign  effort. 

The  primary  changes  reflected  in  the  Model  60  are  more  powerful 
engines,  modified  aerodynamics,  a  3.5  feet  fuselage  extension, 
and  a  larger  fuel  tank.  The  results  were  greater  range,  improved 
aerodynamic  efficiency,  and  increased  payload. 

The  Model  60  development  program  was  originally  intended  to 
involve  merely  a  28  inch  stretch  to  the  Model  55.  [Etherington] 
Just  after  the  program  was  initiated,  the  Federal  Aviation 
Administration  (FAA)  issued  a  change  in  takeoff  requirements  that 
required  Learjet  to  use  a  more  powerful  engine  on  the  Model  60 
than  the  one  used  on  the  Learjet  55.  The  engine  selected,  the 
Garrett-4,  was  an  off-the-shelf  design  being  used  on  a  competing 
business  jet,  the  Citation  7.  The  Garrett-4  would  easily  fit  the 
Model  60  using  the  existing  nacelle  and  pylon  design.  Therefore, 
this  was  considered  a  minor  change  in  requirements  with  only  a 
minimal  impact  to  cost  and  schedule.  [Etherington,  1994] 

Several  months  later,  the  Model  60  design  team  presented  the 
preliminary  design  during  the  first  program  review.  This  meeting 
was  the  sales  department's  first  look  at  plans  for  the  new 
aircraft.  After  the  presentations,  the  vice  president  of  the 
sales  department  stated  that  the  aircraft  would  not  sell;  the 
design  did  not  match  the  desires  of  the  customers  they  had 
recently  surveyed.  The  sales  department  then  provided  new 
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requirements,  primarily  increasing  range  to  a  transcontinental 
distance  and  increasing  luggage  space.  [Etherington,  1994] 

These  changes  had  major  impacts  on  the  design  and  the  program. 

The  new  range  requirement  drove  the  need  for  a  more  powerful 
engine,  greater  fuel  capacity,  and  improved  aerodynamic 
efficiency.  To  accommodate  increased  luggage  space,  the  fuselage 
was  stretched  about  a  foot  more  than  the  originally  planned  28 
inches. 

Instead  of  having  to  convince  an  engine  company  to  develop  a  new 
engine,  Lear jet  engineers  found  an  existing  design  that  met  its 
performance  requirements.  The  PW305  engine  from  Pratt  &  Whitney 
Canada  was  completing  development  and  was  slated  for  use  on  the 
British  Aerospace  1000  business  jet.  Two  of  the  engines  could 
provide  10,450  pounds  of  thrust,  well  above  the  9,200  pounds  of 
thrust  needed.  However,  to  fit  on  the  Model  60,  the  PW305 
required  a  change  in  pylon  design.  [Etherington,  1994] 

To  implement  a  new  pylon  design  without  increasing  drag  an 
unacceptable  amount  was  a  greater  challenge  than  expected. 
[Etherington,  1994]  The  change  also  led  to  modifications  to  the 
wing  leading  edge.  Lear jet  undertook  a  major  effort  using  the 
TRANAIR  computational  fluid  dynamics  (CFD)  code  to  explore  design 
modification  options  and  using  high  speed  wind  tunnel  tests  to 
confirm  drag  predictions.  [Phillips,  1991]  The  new  engine  also 
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required  the  development  of  full  authority  digital  electronic 
control  software  that  would  optimize  power  and  efficiency 
electronically  and  monitor  all  engine  functions. 

The  integrated  avionics  suite  selected  by  Learjet  was  the  Collins 
Pro  Line  System  4.  This  system,  designed  to  minimize  pilot 
workload  and  panel  scanning,  was  essentially  off-the-shelf 
hardware  requiring  minimal  hardware  and  software  modifications. 

The  Model  60  airframe  was  designed  to  last  for  a  time  comparable 
to  that  found  on  other  business  jets.  The  significant  design 
margin  is  evidenced  by  one  of  the  aircraft  being  subjected  to 
50,000  continuous  cabin  pressure  cycles  of  pressures  from  sea 
level  to  51,000  feet.  This  is  equivalent  to  100  years  of  typical 
business  jet  use.  [Brochure,  ca  1994] 

Based  on  sales  projections  from  its  sales  department,  Learjet 
established  its  production  tooling  requirements  for  150  units.  By 
the  first  delivery  in  January  1993,  Learjet  had  33  firm  orders 
(about  a  one  and  a  half  year  backlog)  and  was  expecting  to  be 
producing  Model  60s  at  a  rate  of  18-33  units  per  year  to  at  least 
the  year  2000  if  projected  sales  materialized.  As  of  July  1994, 
Learjet  had  sold  about  fifty  Model  60  aircraft.  The  unit 
production  sales  price  of  one  of  the  aircraft  in  mid-1994  was 
approximately  $9.5  million.  [Etherington,  1994] 
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The  Model  60  development  was  a  company  funded  effort  performed  by 
a  small  team  of  Lear jet  employees  located  at  the  company's 
Wichita,  Kansas,  headquarters.  About  200  people,  including 
subcontractors,  were  involved  in  the  design  and  prototype 
fabrication,  and  they  used  many  of  the  design, 

analysis,  and  manufacturing  tools  used  by  the  aerospace  industry 
in  the  late  1980s  and  early  1990s. 

System  Engineering  Fundamentals 

In  the  development  of  the  Model  60,  the  Lear jet  team  followed 
many  of  the  fundamental  systems  engineering  principles  and 
techniques.  However,  much  of  the  process  was  conducted  with 
minimal  formality.  Given  the  relatively  low  complexity  of  this 
redesign  effort,  such  an  informal  approach  was  adequate  and 
appropriate . 

Some  formality  in  the  development  process  was  evident  in  certain 
controlled  documents.  Specifically,  the  design  requirements  were 
documented  in  a  system  specification  by  the  engineering 
department  after  receiving  the  top  level  customer  desires  from 
the  marketing  department.  The  other  formal  documents  consisted 
primarily  of  a  test  master  plan,  test  plans  and  procedures,  a 
configuration  management  plan,  and  a  quality  plan.  Interface 
control  documents  were  also  maintained  and  controlled.  However, 
no  systems  engineering  management  plan  or  equivalent  was 
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developed  by  Lear jet,  and  neither  were  subsystem  specifications. 
Subsystem  specifications  probably  existed  with  the  subcontractors 
who  had  developed  their  items  independently  of  Lear jet,  but  they 
were  not  generated  as  part  of  the  Model  60  development. 

Given  the  redesign  nature  of  the  development  effort  and  the 
minimal  complexity,  the  Learjet  team  did  not  formulate  a  new  top 
level  model  of  the  aircraft  and  its  subsystems,  nor  did  it 
establish  a  work  breakdown  structure.  However,  it  did  allocate 
requirements  and  define  new  interfaces  as  a  result  of  the 
performance  requirements  and  subsystems  that  were  different  than 
those  of  the  Learjet  Model  55C.  [Etherington,  1994] 

Other  systems  engineering  tasks  of  the  design  process  were  also 
somewhat  informal.  While  modeling  and  analysis  tools  were  used  to 
assess  and  refine  the  design,  primarily  in  the  area  of 
aerodynamics,  the  team  never  performed  a  formal  tradeoff  analysis 
of  options.  Nor  was  a  risk  analysis  performed.  [Etherington, 

1994]  Instead  of  beginning  from  a  clean  slate,  the  existing 
Model  55C  was  deemed  the  starting  point  of  development  with  the 
expectation  that  the  end  result  would  be  merely  a  modification. 
The  approaches  to  meet  the  performance  requirements,  with  the 
possible  exception  of  the  aerodynamic  refinements,  were  rather 
straight  forward  given  the  initial  design  constraints. 

Consistent  with  its  stated  company  approach,  Learjet  built  the 
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Model  60  airframe  in-house  and  obtained  most  of  the  subsystems 
from  elsewhere.  This  is  reasonable  since  Lear jet  is  not  a  large 
aerospace  company,  and  it  does  not  possess  the  resources  in 
people,  facilities,  and  experience  to  fabricate  everything  itself 
in  an  economical  manner.  On  this  fact  alone,  Lear jet  would  be 
expected  to  conduct  no  more  than  minimal  make-or-buy  evaluations 
since  the  make-or-buy  decisions  were  clear  cut  on  most  items. 
Furthermore,  since  the  Model  60  uses  a  majority  of  the  same 
subsystems  and  components  as  the  Model  55,  most  of  the  parts  and 
suppliers  were  already  set.  Despite  only  limited  evaluations,  the 
Model  60  make-or-by  decision  performance  based  on  Learjet's 
policy  was  reasonable. 

Since  the  Model  60  is  a  direct  redesign,  there  was  not  a 
significant  issue  in  the  beginning  with  regards  to  being  able  to 
technically  develop  it.  The  program,  however,  never  performed  a 
validation  audit  of  its  requirements.  [Etherington,  1994] 

Instead,  Lear jet  appears  to  have  relied  primarily  on  the  market 
surveys  early  in  the  program  that  indicated  that  potential 
customers  were  interested  in  increased  range  and  payload.  The 
Model  60  team  also  performed  extensive  simulation  testing  and 
analysis  to  validate  the  aerodynamics  requirements  of  the 
aircraft . 

Learjet  followed  verification  and  integrated  testing  in  a  manner 
required  for  FAA  certification.  Therefore,  most  of  the 
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verification  and  integrated  testing  requirements  were  known  at 
the  beginning  of  the  effort.  The  program  created  and  flew  one 
engineering  prototype  before  completing  three  deliverable 
preproduction  units  that  were  flown  in  the  flight  certification 
test  program.  Based  partly  on  the  results  of  the  engineering 
prototype  flight  test,  additional  refinements  were  made  to  the 
design  to  improve  range,  handling,  and  center  of  gravity  limits. 
[Aviation  Week,  1991]  These  changes  were  retrofitted  into  the 
engineering  prototype  and  incorporated  into  the  four 
preproduction  prototype  aircraft.  No  major  problems  hampered  the 
completion  of  development  from  that  point  to  production,  and  no 
major  manufacturing  or  quality  problems  developed.  The  only  issue 
of  note  was  the  need  for  minor  modifications  to  the  environmental 
control  system.  Aside  from  the  varying  options  ordered  by 
different  customers,  the  basic  aircraft  configuration  has 
remained  highly  stable  from  the  four  preproduction  prototypes 
through  production.  [Etherington,  1994]  Furthermore,  the  four 
preproduction  units  were  delivered  to  customers  after  the 
successful  completion  of  the  18— month  flight  test  program  and  FAA 
certification . 

The  off-the-shelf  integrated  avionics  suite  simplified  the 
integration  task  for  Learjet  by  reducing  the  interfaces  Learjet 
had  to  develop.  The  avionics  supplier,  Collins,  was  responsible 
to  Learjet  for  ensuring  that  the  flight  management  system 
computer,  the  navigation  and  guidance  units,  the  weather  radar. 


192 


and  the  cockpit  displays  all  worked  together  when  installed  in 
the  airframe.  Purchasing  the  entire  avionics  suite  from  one 
source  reduced  the  need  for  extensive  ground  avionics  integration 
testing  by  Lear jet.  Also,  with  the  engine  already  having  been 
developed  and  flight  tested  by  Pratt  &  Whitney  Canada  on  a 
different  test  aircraft,  the  first  engines  arrived  at  Lear jet  as 
developed  and  tested  units.  Therefore,  Lear jet's  developmental 
testing  responsibility  was  primarily  system-level  using 
engineering  and  production  prototype  flight  units  to  demonstrate 
integrated  performance  as  part  of  FAA  certification  requirements. 

The  changes  in  requirements,  interfaces,  and  configuration  that 
occurred  throughout  FSED  were  all  controlled  adequately  by 
Lear jet.  while  the  program  did  not  have  a  full-blown 
configuration  management  system,  it  did  have  a  formal  change 
control  system  with  a  change  control  board.  [Etherington,  1994] 

Since  Lear jet  is  a  small,  manufacturing  oriented  company,  the 
production  organization  was  involved  early  in  the  program  to  help 
in  planning  for  production.  Since  the  Lear jet  was  a  redesign  of  a 
successful  aircraft,  there  was  not  a  strong  motivation  to  review 
and  possibly  revise  the  manufacturing  processes  for  the  Model  60 
to  improve  ease  of  manufacturing  and  reduce  cost.  Learjet, 
however,  used  some  computer  aided  design  (CAD)  partially 
integrated  with  advanced  numerically  controlled  milling  machine 
tools  for  manufacturing  airframe  components.  The  fabrication  of 
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the  Model  60  did  not,  however,  require  the  development  of 
advanced,  unique  manufacturinq  processes. 

Even  though  Lear jet  was  not  involved  in  developing  the  avionics 
suite  and  propulsion  units,  the  Model  60  team  had  to  manage  the 
avionics  integration  into  the  airframe,  as  well  as  the  interfaces 
between  the  engines  and  airframe.  The  most  critical  interfaces 
between  subsystems  that  Learjet  had  to  directly  control  were 
those  between  the  avionics  and  the  engines.  This  was  done  with 
extensive  involvement  by  the  engine  and  avionics  subcontractors. 

The  management  of  the  systems  integration  as  well  as  the  entire 
technical  effort  was  done  in  a  manner  consistent  with  a  small, 
cohesive  technical  team.  Consequently,  the  only  formal  reviews 
held  were  the  preliminary  design  review  and  software  walk¬ 
throughs  on  the  digital  engine  control  software.  These  were  in 
addition  to  the  weekly  engineering  meetings  involving  primarily 
Learjet  team  members.  There  were  limited  forums  for  formal  review 
of  the  design,  but  they  do  not  appear  to  have  been  needed  since 
the  customers  were  not  involved  in  development. 

Life  cycle  considerations  did  not  require  a  major  amount  of 
attention  from  the  design  team.  The  ground  support  equipment, 
maintenance  procedures,  and  the  training  were  the  same  as  for  the 
Model  55,  with  the  exception  of  support  elements  dealing  with  the 
new  engine.  Furthermore,  the  Learjet  design  team  appears  to  have 
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sought  to  maintain  mission  reliability  at  the  level  of  the  Model 
55,  already  very  high,  rather  than  to  push  for  marginal 
improvements.  However,  the  Model  60  does  incorporate  a  key 
supportability  feature  to  enhance  maintenance  operations,  which 
is  a  built-in  diagnostic  system  in  the  engines  to  record 
performance  data. 

In  order  to  plan  and  track  the  progress  of  the  Model  60 
development  effort,  the  program  manager  used  Gantt  chart 
scheduling  and  a  computerized  cost  accounting  system.  While  not 
particularly  sophisticated  tools,  they  were  adequate  for  the 
relatively  low  complexity  level  of  the  effort.  The  program 
manager,  however,  did  not  convene  regular  program  reviews,  which 
was  indicative  of  the  informal  nature  of  conducting  the 
development . 

Development  Environment 

After  the  requirements  changes  early  in  FSED,  the  development 
environment  was  stable  due  to  consistent  funding,  stable 
requirements,  low  personnel  turnover,  and  reasonably  high 
configuration  stability.  The  Model  60  development  was  fully 
funded  by  Lear jet  and  FSED  was  initiated  even  before  firm  orders 
were  received.  The  company  funded  the  effort  at  a  rate  of  about 
$30  million  per  year.  Lear jet  had  made  a  strong  commitment  to 
develop  the  aircraft  and  did  not  constrain  funding  flow,  reduce 


195 


the  workforce,  or  threaten  to  cancel  the  effort  as  the  cost 
estimates  increased  due  to  the  changes  in  scope. 

Designers  used  a  3-dimensional  wire-frame  computer  aided  design 
(CAD)  program  called  UG  II  for  designing  the  portions  of  the 
Model  60  that  were  different  from  the  Model  55.  In  addition  to 
the  use  of  a  CFD  program  with  a  NASA  Cray  computer  and  wind 
tunnels  to  verify  CFD  results  as  mentioned  previously,  the  design 
team  utilized  the  NASTRAN  finite  element  program  for  mechanical 
analysis  and  AutoCAD  for  minor  wiring  diagram  changes.  A  cabin 
mockup  was  built  to  show  cabin  modifications,  and  one  engineering 
prototype  flight  unit  was  fabricated  and  flown. 

The  Model  60  was  developed  in  a  close-knit  project  team 
environment.  The  relatively  small  airframe  and  integration 
engineering  team  was  physically  located  together,  and  the 
manufacturing  personnel  were  on  the  same  grounds.  Close 
cooperation  existed  between  the  Lear jet  personnel  and  those  from 
the  major  subcontractors.  However,  Lear jet  did  not  oversee  the 
development  of  the  major  subsystems,  especially  since  they  were 
already  developed  by  the  time  the  program  began.  Neither  did  the 
development  team  have  any  direct  involvement  with  the  customers, 
most  of  whom  were  corporations.  That  task  was  left  to  the  sales 
department,  and  it  appears  that  interaction  between  the  top-level 
requirements  developers  and  the  design  team  was  limited. 
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The  limited  complexity  of  the  program,  the  small  size  of  the 
development  team,  and  the  physical  proximity  of  the  Learjet 
members  with  each  other  enabled  adequate  communication  within  the 
technical  team,  thereby  precluding  the  need  for  more  formal 
systems  engineering  activities  and  techniques-  Furthermore,  the 
maturity  of  the  major  subsystems  lessened  the  degree  of  Lear jet's 
oversight  activities  over  the  subcontractors  and  minimized 
potential  communication  problems. 

The  Model  60  was  designed  to  attain  specific  performance  on  a 
variety  of  parameters,  the  primary  one  being  range-  The  warranty, 
however,  does  not  address  it  or  other  performance  parameters 
specifically.  Instead,  the  warranty  covers  the  failure  of  parts 
built  by  Learjet  that  occur  within  five  years  after  delivery  and 
the  failure  of  parts  manufactured  by  subcontractors  and  vendors 
that  occur  within  two  years  after  delivery. 

Summary /Conclus ion 

The  Model  60  aircraft  design  attained  the  range  and  capacity 
performance  desired  by  the  business  jet  market  and  specified  by 
the  Learjet  sales  department.  In  achieving  this  success,  the 
development  team  performed  many  of  the  fundamental  practices  of 
systems  engineering  in  an  informal  manner  on  this  primarily  low- 
risk  redesign  effort.  While  the  development  environment  was 
reasonably  stable  in  most  areas,  the  requirements  development 


process  in  the  beginning  of  the  program  was  not  conducted 
effectively.  The  consequences  of  the  changes  in  key  requirements 
after  program  initiation  were  increased  scope,  delay  in  schedule, 
and  cost  overrun.  Despite  these  shortcomings  of  the  development 
effort,  the  Model  60  met  its  key  performance  requirements  and 
satisfied  its  customers. 


Table  3.5-1  Model  60  Performance  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

TECHNICAL  PERFORMANCE  - 
INITIAL 

0-10 

1 

9 

9 

TECHNICAL  PERFORMANCE  - 
MATURE 

0-10 

1 

10 

10 

COST  PERFORMANCE 

0-10 

1 

2 

2 

SCHEDULE  PERFORMANCE 

0-10 

1 

4 

4 

PERFORMANCE  TOTAL 
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Table  3.5-2  Model  60  Systems  Engineering  Fundamentals  scores. 


Table  3.5-3  Model  60  Development  Environment  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

EMPHASIS  ON  THE  CUSTOMER 

0-10 

1 

6 

6 

STABILITY  OF  REQUIREMENTS 

AND  CONFIGURATION 

0-10 

1 

3 

3 

FUNDING  AND  WORKFORCE-LEVEL 
STABILITY 

0-10 

1 

8 

8 

STRONG  SUPPORT  FOR  PROGRAM 

0-10 

1 

8 

8 

CONTINUITY  OF  CORE 
DEVELOPMENT  TEAM 

0-10 

1 

10 

10 

STABILITY  OF  ORGANIZATIONAL 
STRUCTURE 

0-10 

1 

9 

9 

COOPERATION  AMONG  ALL 
STAKEHOLDERS 

0-10 

1 

8 

8 

EFFECTIVE  COMMUNICATION 
WITHIN  TEAM 

0-10 

1 

5 

5 

FLEXIBILITY  AND  AUTONOMY 

0-10 

1 

8 

8 

WORKFORCE  EXPERTISE  AND 
EXPERIENCE 

0-10 

1 

8 

8 

ACCOUNTABILITY  FOR  SYSTEM 
PERFORMANCE 

0-10 

1 

6 

6 

DEVELOPMENT  ENVIRONMENT 

IIIIIIIII 

iaiiiii 

79 

TOTAL 

’ 

200 


Table  3.5-4  Model  60  Design  Difficulty  scores 


Elements 

Range 

Score 

TYPE 

0-15 

3 

KNOWLEDGE  COMPLEXITY 

0-10 

4 

STEPS 

0-10 

6 

QUALITY  IMPLEMENTATION 
EFFORT 

0-10 

5 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

3 

SELLING  PRICE  CONSTRAINT 

0-5 

4 

DESIGN  DIFFICULTY  TOTAL 

0-55 

25 

Table  3.5-5  Model  60  Resources  scores. 


Elements 

Range 

Score 

COST 

0-15 

9 

TIME 

0-10 

5 

INFRASTRUCTURE 

0-10 

6 

RESOURCES  TOTAL 

0-35 

20 
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3.6  CASE  STUDY:  MCDONNELL  DOUGLAS  MD-11  COMMERCIAL  TRANSPORT 

The  MD-11  is  a  large,  long-range,  three-engine  commercial 
transport  developed  in  the  late  1980s  and  early  1990s  by  the 
Douglas  Aircraft  Company  of  the  McDonnell  Douglas  Corporation  for 
commercial  passenger  and  cargo  travel.  The  MD-11  was  intended  to 
fit  into  a  market  segment  between  the  large  capacity  Boeing  747 
and  the  medium  capacity  Boeing  767  and  Airbus  Industrie  A300  and 
A330.  The  MD-11 's  direct  competitors  are  the  four-engine  Airbus 
A340  and  the  new  twin-engine  Boeing  777.  Douglas  initially 
approached  the  MD-11  development  as  a  relatively  simple,  low  risk 
modification  of  the  DC-10.  The  effort,  however,  quickly  became 
significantly  more  complicated  as  unanticipated  systems  issues 
appeared.  As  a  consequence,  the  program  was  a  difficult 
experience.  However,  despite  problems  during  development  and 
initial  customer  operation,  the  MD-11  was  eventually  developed 
into  a  successful  product. 

Development  History,  Design,  and  Performance 

McDonell  Douglas  launched  the  MD-11  full-scale  engineering 
development  and  production  (FSED)  program  on  December  30,  1986, 
with  firm  orders  from  twelve  airlines  for  52  aircraft  and  options 
for  40  more.  The  first  flight  took  place  three  years  later  on  10 
January  1990.  The  flight  test  program  involved  five  aircraft  for 
ten  months,  allowing  the  first  customer  delivery  in  late  1990. 


203 


(See  Figure  3.6.)  Up  until  flight  testing,  the  development 
program  was  nearly  a  year  behind  original  schedule  due  to  a 
variety  of  problems  at  Douglas.  These  included  overly  optimistic 
schedules,  delays  in  receiving  parts,  sweeping  management  and 
production  changes  attempted  in  the  middle  of  development,  and 
various  technical  difficulties.  [Aviation  Week,  1989]  However, 
the  company  reduced  the  schedule  delay  during  flight  testing,  and 
the  first  customer  delivery  of  a  certified  aircraft  was  nine 
months  (23  percent)  late. 

The  $722  million  development  effort  was  funded  entirely  by 
McDonnell  Douglas,  and  it  experienced  an  overrun  of  about  30 
percent  over  the  original  baseline  cost.  [Larson,  1995]  By  late 
1994,  122  MD-lls  had  been  delivered,  51  units  were  in  various 
stages  of  fabrication,  and  88  were  on  option  or  reserve  order. 
This  is  in  excess  of  the  program's  break  even  point  for  MD-11 
deliveries,  thereby  making  the  MD-11  a  profitable  venture. 

[Larson,  1995]  The  1994  selling  price  for  each  of  the  initial, 
shorter  range  versions  of  the  MD— 11  with  standard  training  and 
maintenance  manuals  is  about  $116  million. 

The  MD-11  is  a  stretched  and  updated  version  of  the  DC-10,  and  it 
therefore  can  be  categorized  as  a  redesign.  The  initial  version 
was  designed  to  carry  nearly  300  passengers  a  distance  of  almost 
7,000  miles.  The  basic  airframe  is  the  same  as  the  DC-10,  but  the 
fuselage  is  18  feet  6  inches  longer,  aerodynamics 
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Figure  3.6  MD-11  development  schedule. 


1985  1986  1987  1988  1989  1990 

Reqmts  &  Concept  Dev. 

FSED 

1st  Flight  ^ 

Flight  Testing  (Certification) 

1st  Production  Delivery  ^ 


are  improved,  more  powerful  engines  are  used,  and  a  new  highly 
integrated  digital  avionics  system  is  incorporated.  The  MD-11 
design  philosophy  was  to  use  a  high  level  of  automation  to  take 
care  of  routine  flying  tasks  as  well  as  emergency  procedures 
automatically,  thereby  enabling  two  pilots  to  handle  the  workload 
without  a  flight  engineer.  As  a  consequence,  the  cockpit  has 
sixty  percent  fewer  switches,  gauges,  and  lights  than  the  DC— 10. 
[Aviation  Week,  1990a] 

The  fact  that  the  MD-11  was  a  derivative,  or  redesign,  of  the  DC- 
10  helped  reduce  development  risk.  However,  major  technical 
challenges  still  existed  that  Douglas  did  not  anticipate.  Of 
primary  significance  was  the  sophisticated  new  digital  avionics 
system  which  impacted  every  subsystem  on  the  aircraft.  [Smith, 
1990b]  This  dual  flight  management  system  allows  automatic 
operation  of  the  aircraft  fuel,  air,  electrical,  and  hydraulic 
systems,  and  it  also  incorporates  the  automatic  flight  system. 


205 


Each  of  these  subsystems  is  controlled  by  a  pair  of  dedicated 
computers.  The  subcontractor  Honeywell  was  responsible  for 
supplying  and  integrating  all  the  avionics  for  the  program,  a 
role  normally  filled  by  the  aircraft  contractor.  Even  though 
Douglas  personnel  oversaw  the  complex  avionics  integration  effort 
instead  of  performing  it,  Douglas  still  had  to  integrate  the 
avionics  with  the  rest  of  the  aircraft.  This  required  developing 
new  interface  devices  for  the  analog  DC-10  components  that  were 
carried  over  into  the  MD-11  design  as  well  as  for  new  subsystems. 
From  Douglas'  point  of  view,  its  avionics  challenge  was  therefore 
mainly  that  of  software  integration,  since  that  was  the  major 
area  with  which  they  had  little  knowledge  or  experience.  [Smith, 
1990b]  The  software  problems  turned  out  to  be  more  difficult  to 
deal  with  than  expected. 

The  MD-11  development  team  also  faced  a  significant  challenge  to 
improve  aerodynamic  performance.  The  design  target  was  to  reduce 
drag  by  eight  percent  compared  with  the  DC-10.  [Aviation  Week, 
1990b]  To  achieve  this,  the  designers  included  improved 
aerodynamic  features  such  as  winglets,  a  redesigned  wing  trailing 
edge,  a  smaller  horizontal  tail  with  integral  fuel  tanks,  and  an 
extended  tail  cone.  A  considerable  amount  of  wind  tunnel  testing 
was  conducted  to  verify  and  refine  the  design.  At  the  time  of  the 
first  aircraft  deliveries,  drag  reduction  was  short  of  the  goal. 
Nominal  aerodynamic  performance,  however,  was  reached  in  1991  and 
retrofitted  into  completed  aircraft. 
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While  Douglas  anticipated  some  difficulties  with  improving 
aerodynamic  design,  it  didn't  expect  many  hardware  problems.  One 
such  problem  that  did  arise  impacted  key  performance.  The  fuel 
consumption  performance  for  the  engines  from  Pratt  &  Whitney  and 
General  Electric,  both  which  are  upgraded  versions  of  established 
propulsion  units  used  on  other  aircraft,  were  below  design 
requirements  by  amounts  from  4.5-6  percent  for  the  initial  units 
delivered  to  Douglas.  [Smith,  1990c]  Although  the  engine 
subcontractors  improved  fuel  consumption  efficiency  throughout 
development  and  production,  the  original  allocated  performance 
goal  was  never  achieved.  [Larson,  1995] 

Compounding  the  engine  shortfalls  was  another  problem  common  to 
most  aircraft  programs,  weight  growth.  The  earliest  MD-lls  were 
about  4,000  lb.,  or  one  and  a  half  percent,  above  projected  empty 
weight.  The  extra  weight  was  a  result  of  changes  in  preliminary 
load  figures,  higher  than  expected  weights  for  the  interior 
configuration,  some  new  MD-11  components  from  subcontractors,  and 
configuration  changes  requested  by  customers.  [Smith,  1990b]  The 
result  of  the  overweight  condition  coupled  with  the  lower  engine 
efficiencies  and  the  less  than  expected  improvements  to 
aerodynamics  was  that  the  initial  MD-lls  did  not  meet  the 
promised  payload/range  performance.  This  was  of  critical 
importance  since  many  of  the  customers  were  drawn  to  the 
advertised  MD-11  payload/range  capability  which  enabled  them  to 
fly  profitable  non-stop  international  flights.  Furthermore , 
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Douglas  was  contractually  liable  for  payload/range  capacity. 
Although  Douglas  eventually  resolved  most  of  the  shortfall 
primarily  through  aerodynamic  improvements  and  weight  reductions, 
the  initial  aircraft  delivered  were  not  in  compliance  and  had  to 
be  modified  once  fixes  were  identified.  At  four  years  into 
production,  the  MD-lls  coming  off  the  production  line  are  meeting 
all  performance  guarantees.  [Larson,  1995] 

Given  the  high  level  of  commonality  between  the  MD-11  and  the  DC- 
10  and  the  fact  that  the  DC-10  production  line  was  ending  at  the 
same  time  the  MD-11  fabrication  was  to  begin,  manufacturing 
planned  to  refurbish  most  of  the  DC-10  tooling  to  original 
condition  for  MD— 11  use.  The  rest  of  the  needed  tooling,  about 
twenty  percent  of  the  total,  were  either  modifications  or  new 
purchases.  [Aviation  Week,  1987] 

With  the  commonality  between  the  MD-11  and  DC- 10  designs  and 
production  lines,  it  is  not  surprising  that  the  MD-11  experienced 
the  same  type  of  manufacturing  problems,  primarily  inefficiency 
on  the  production  floor.  This  situation  was  indicated  by  a  high 
degree  of  rework  and  repair  and  was  due  to  quality  related 
problems.  Another  measure  of  inefficiency  is  the  percentage  of 
drawing  rereleases  from  the  initial  drawings  for  an  aircraft  on 
the  production  floor.  These  drawing  changes  are  a  result  of 
design  updates  due  to  errors  as  well  as  customization  features 
requested  by  the  customer,  and  a  40  percent  rate  was  not  uncommon 
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at  Douglas.  Such  errors  are  expensive.  The  administrative  cost  of 
processing  these  drawing  rereleases,  or  engineering  orders,  is 
about  $4000  per  drawing,  and  the  initial  MD-lls  built  had  from 
approximately  100  to  350  engineering  orders  against  each. 

[Larson,  1995] 

In  order  to  improve  the  way  it  did  business  and  reduce 
manufacturing  costs,  McDonnell  Douglas  tried  to  implement  a 
product  teaming  approach  which  it  called  the  Total  Quality 
Management  System  (TQMS)  throughout  the  entire  corporation  in 
1989,  including  Douglas.  This  effort  restructured  Douglas's 
entire  way  of  developing  aircraft,  thereby  replacing  the 
traditional  functional  and  hierarchical  management  approach.  This 
transformation,  however,  was  not  carried  out  effectively. 
Furthermore,  anticipated  improvements  have  not  yet  become 
tangible  as  high  error  rates  are  still  seen  on  the  newest  MD-lls 
coming  off  the  production  line.  [Larson,  1995] 

While  the  MD-11  design  has  its  problems,  it  does  possess  a 
certain  flexibility  and  robustness.  From  the  outset  of  the 
program,  there  were  four  versions  of  the  MD— 11  planned.  These 
consisted  of  the  standard  MD-11,  the  longer  range  MD-11ER  with  a 
standard  DC-10  length  fuselage,  the  MD-11  combination  passenger/ 
freight  transport,  and  the  MD-11F  all-cargo  aircraft.  Later,  a 
stretched  version  of  the  MD-11,  the  MD-12X,  was  also  considered. 
And  finally,  Douglas  is  looking  at  a  possible  twin-engine  version 
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of  the  MD-11. 

This  flexibility  is  due  partly  to  the  structural  margin  included 
in  the  basic  airframe  design.  Douglas  Aircraft  Company  airplanes 
have  the  best  structural  integrity  of  the  large  airliners. 
[Larson,  1995]  They  have  traditionally  been  overdesigned  for 
safety  reasons  as  part  of  a  conservative  design  approach.  The 
company,  therefore,  has  not  been  inclined  to  embrace  new, 
lightweight  structural  materials  as  much  as  it  competitors.  For 
example,  the  MD-11  has  a  somewhat  low  overall  composite  content 
by  weight  of  3.7  percent  compared  to  about  9  percent  for  the 
Boeing  777.  The  added  structural  weight  contributes  to  longer 
airframe  life.  The  projected  design  life  of  the  MD-11  airframe  is 
60,000  flight  hours,  or  20,000  flights.  (Airplane  wear  is  a 
function  of  cycles,  not  flight  time.)  The  aircraft  can  fly  for 
longer  periods,  but  Federal  Aviation  Administration  (FAA) 
requires  more  inspections  after  60,000  hours.  Having  greater 
structural  integrity  can  extend  useful  operating  life  beyond  the 
projected  life,  and  this  makes  a  Douglas  airplane  design  better 
for  modification  and  re— engining.  As  a  result  of  this  structural 
robustness,  Douglas  aircraft  generally  have  greater  resale  value 
than  its  competitors.  [Larson,  1995] 

While  airlines  and  air  freight  carriers  may  consider  long  term 
design  flexibility  and  resale  value  as  an  element  in  customer 
satisfaction,  operating  performance  is  the  fundamental 
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determinant.  One  example  of  this  deals  with  the  MD-ll's 
controllability.  According  to  a  Douglas  engineering  manager,  the 
DC-10  has  a  reputation  for  being  an  excellent  pilot's  airplane. 
[Larson]  Therefore,  the  MD-ll's  controllability  was  judged 
against  the  DC-10.  The  FAA  and  Douglas  pilots  who  flew  the  MD-11 
during  certification  flights  agree  that  the  aircraft's  handling 
qualities  and  control  harmony  are  equal  to  or  better  than  those 
of  the  DC-10.  [Scott,  1990] 

While  payload/range  performance  has  always  been  superior  to  the 
DC-10,  the  initial  MD-lls  were  not  meeting  the  requirement.  Over 
a  period  of  two  years,  Douglas  developed  improvements  and  had 
them  retrofit  into  the  delivered  aircraft.  In  the  meantime, 
though,  the  airlines  had  less  capable  aircraft  to  operate. 

The  performance  of  the  initial  MD-lls  was  marginal  in  other  areas 
as  well.  One  example  deals  with  the  experience  by  one  of  the 
first  MD-11  customers,  Swissair.  The  airline  experienced  numerous 
minor,  but  frequent  and  annoying  mechanical  problems  during  the 
initial  months  of  operation  involving  the  fuel  system  and 
passenger  comfort  and  entertainment  systems.  In  spite  of  the  many 
problems,  Swissair  claims  it  experienced  fewer  overall  early 
phase-in  problems  with  the  MD-11  than  for  other  aircraft  for 
which  it  was  an  initial,  or  launch,  customer.  [Lenorovitz,  1991] 
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American  Airlines,  another  early  customer,  was  greatly  displeased 
with  the  MD-11 's  initial  performance.  Like  Swissair,  American  had 
to  undergo  a  difficult  period  of  aircraft  introduction.  After 
Douglas  spent  a  considerable  amount  of  resources  correcting 
problems,  the  airline  became  satisfied  with  the  MD-11.  Now  that 
the  primary  problems  have  been  worked  out  by  Douglas,  the 
aircraft  in  operation  and  those  coming  off  the  production  line 
exceed  stated  performance  requirements  in  almost  all  categories. 
[Larson,  1995] 

The  MD-11  has  a  total  of  434,970  parts,  not  including  nuts  and 
bolts.  With  a  system  this  large  and  complex,  some  problems  can  be 
expected  for  the  first  units  regardless  of  how  well  the 
development  is  carried  out.  However,  most  of  the  MD-11  problems 
could  have  been  prevented  had  Douglas  not  underestimated  the 
integration  challenge. 

Systems  Engineering  Fundamentals 

Over  the  past  thirty  years,  Douglas  Aircraft  Company,  which  is 
one  of  the  world's  largest  developers  and  manufacturers  of 
commercial  passenger  jet  aircraft,  has  produced  a  series  of 
successful  commercial  transports,  including  the  DC-8,  DC-9,  DC- 
10,  and  MD-80 .  During  this  time,  it  had  developed  its  own  version 
of  a  traditional  approach  to  aircraft  development.  It  was 
essentially  a  sequential  process  by  which  engineers  drove  the 
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aircraft  design  with  minimal  influence  by  non-engineering 
organizations.  Manufacturing  was  involved  during  the  design  phase 
in  order  to  plan  for  fabrication,  but  it  was  not  in  a  role  to 
significantly  influence  the  design  itself.  Furthermore,  the 
Douglas  approach  was  further  identified  by  a  breakout  of  the 
technical  and  non-technical  specialties  under  separate  and  highly 
functionally  oriented  organizations.  This  approach,  with  its 
attendant  weaknesses,  was  in  place  at  the  beginning  of  the  MD-11 
development  effort. 

During  the  development  of  the  MD-11,  the  company  possessed 
aerospace  and  systems  development  tools  standard  for  the  mid- 
1980s,  such  as  computer  aided  design  and  drafting,  mechanical 
analysis  programs,  computer  simulations,  and  wind  tunnels.  It  was 
therefore  adequately  equipped  to  conduct  a  redesign  effort. 

The  primary  market  potential  for  the  MD-11  was  as  a  replacement 
for  the  DC-lOs  and  Lockheed  L-lOlls  flying  transcontinental  and 
international  routes.  Since  top  McDonnell  Douglas  and  Douglas 
Aircraft  Company  management  viewed  the  MD-11  as  a  relatively 
straightforward  modification  of  an  existing  design,  there  was 
little  emphasis  on  developing  a  new  set  of  system  requirements. 

As  a  result,  no  integrated  requirements  document  was  developed  at 
the  beginning  of  the  program.  Instead,  Douglas  used  the  existing 
DC-10  specification  and  the  standard  commercial  aircraft  design 
specification,  DS-1100.  [Larson,  1995]  Douglas's  approach  to 
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supplementing  these  old  requirements  documents  was  to  use 
marketing  surveys  to  refine  higher  level  requirements.  However, 
concerning  detail  design  issues,  the  development  team  made 
assumptions  as  to  what  the  airlines  would  want.  [Larson,  1995] 
There  was  one  exception.  Douglas  involved  pilots  from  thirty- 
seven  airlines  in  the  development  of  the  MD-11  cockpit. 

The  detail  requirements,  although  incompletely  defined  by  the  MD- 
11  designers  and  inadequately  flowed  down  to  lower-tier 
suppliers,  were  allocated  to  closely  match  DC-10  system  model. 
Furthermore,  while  many  of  the  subsystems  did  not  change  much, 
the  interfaces  between  many  of  them  did.  This  was  due  in  large 
part  to  the  new  integrated  avionics  system  that  tied  together 
many  of  the  aircraft's  functions. 

Douglas  had  delegated  integrated  avionics  development 
responsibilities  to  its  subcontractor,  Honeywell,  and  the  engine 
development  was  the  responsibility  of  Pratt  &  Whitney  and  General 
Electric.  Furthermore,  the  company  sought  to  maintain  a  high 
degree  of  design  and  manufacturing  process  commonality  with  the 
DC-10  in  order  to  reduce  the  MD-11  development  risks  and  costs. 
Nevertheless,  alternative  subsystems  in  areas  directly  under 
Douglas's  design  control  were  considered  to  a  limited  extent 
throughout  the  aircraft.  Two  of  the  key  areas  where  Douglas 
performed  considerable  tradeoff  analysis  were  aerodynamics  and 
structural  weight,  both  in  order  to  improve  fuel  efficiency  and 
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achieve  the  payload/range  performance  requirements. 

Analysis  had  also  gone  into  determining  whether  to  develop  and 
build  a  subsystem  or  component  in-house,  or  buy  it  from  outside 
the  company.  There  are,  however,  several  major  components  that 
Douglas  does  not  even  consider  fabricating  itself.  Its  role  is 
primarily  that  of  airframe  manufacturer  and  integrator,  and 
therefore  it  purchases  engines  and  avionics  from  companies 
specialized  in  those  areas.  Regarding  the  airframe  and  the  rest 
of  the  aircraft,  the  large  number  of  subcontractors  and 
suppliers,  both  national  and  international,  attest  to  a  high 
level  of  consideration  going  into  the  make-or-buy  decision 
process. 

Validating  the  MD-11  requirements  was  another  activity  performed 
by  Douglas,  though  not  necessarily  in  a  formal  manner.  Since  the 
MD-11  was  a  redesign  that  did  not  require  the  development  of  new 
technologies,  it  was  obvious  to  Douglas  that  the  system  could  be 
built.  Furthermore,  through  the  use  of  computer  simulations  and 
wind  tunnel  testing,  Douglas  was  confident  that  the  aircraft  it 
was  designing  would  meet  the  customers'  top-level  requirements  as 
understood  from  the  marketing  surveys  and  discussions  with  the 
airlines  that  had  placed  firm  orders.  However,  it  did  not 
accomplish  formal  requirements  audits. 


As  systems  integrator  with  a  large  number  of  subcontractors  and 
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suppliers,  Douglas  was  focused  more  on  verifying  performance  at 
the  system-level  rather  than  at  the  component  and  subsystem- 
level.  However,  during  FSED,  it  participated  and  paid  close 
attention  to  the  tests  performed  by  subcontractors.  This  was 
especially  true  for  the  engine  tests. 

In-house,  Douglas  used  a  variety  of  tools  to  assist  in 
verification  activities.  It  built  and  tested  wind  tunnel  models 
to  refine  the  aerodynamic  design.  A  DC-10  fuselage  was  turned 
into  a  MD-11  development  mockup,  and  it  was  used  for  checking 
mechanical  fit  of  interior  components  and  potential  cabin 
configurations.  No  engineering  prototypes  were  built,  though. 

The  first  units  produced  were  relatively  mature  configurations, 
and  they  were  subjected  to  a  full  range  of  inspections  and  tests. 
Furthermore,  they  were  delivered  to  customers  after  modifications 
at  the  end  of  flight  testing. 

Although  avionics  integration  was  performed  primarily  by  the 
subcontractor  Honeywell,  Douglas  representatives  participated  in 
and  conducted  some  of  the  testing.  Due  to  the  critical  importance 
of  the  avionics  to  the  entire  aircraft,  a  large  amount  of 
avionics  ground  testing  and  software  checks  had  been  planned. 

Some  of  these  tests  were  performed  on  a  flight  deck  simulator 
that  was  run  by  actual  aircraft  computers.  These  activities  were 
accomplished  well  in  advance  of  the  first  flight  test,  with  some 
of  them  in  the  Douglas  avionics  test  center. 
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Although  considerable  ground  avionics  component  checkout  by 
Douglas  occurred  during  development,  the  production  verification 
approach  was  very  different.  During  production,  Douglas  installed 
avionics  units  without  bench  testing  and  returned  any  faulty  ones 
to  the  manufacturer.  This  placed  the  avionics  component 
verification  burden  on  Honeywell,  and  it  allowed  Douglas  to 
eliminate  its  avionics  test  center.  [Aviation  Week,  1987] 

The  MD-11  was  subjected  to  a  comprehensive  series  of  integrated 
flight  tests  in  accordance  with  a  comprehensive  test  and 
evaluation  master  plan  leading  to  certification  by  the  FAA.  In 
preparation  of  testing,  350  miles  of  test  wiring  and  more  than 
1000  remote  sensors  were  installed  during  assembly  of  the  first 
aircraft  so  that  8,000  different  temperature,  pressure, 
acceleration,  and  stress  measurements  could  be  recorded.  [Kubel, 
1991]  When  completed,  this  MD-11  was  subjected  to  ground 
vibration  tests  to  demonstrate  that  the  airframe  had  sufficient 
structural  damping  characteristics.  It  passed  with  no  problems. 

Five  of  the  first  production  MD-lls  were  used  for  FAA  flight 
certification  testing.  While  originally  planning  for  only  three 
aircraft,  Douglas  added  two  more  to  minimize  a  slip  in  the 
overall  program  schedule.  Two  of  the  units  were  focused  mainly  on 
avionics  testing,  while  the  other  three  were  primarily  dedicated 
to  aerodynamic,  engine,  functional  and  reliability  testing. 
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Douglas  performed  interface  and  design  configuration  status 
accounting  and  control  with  varying  degrees  of  success. 

Most  of  the  initial  MD-11  design  was  placed  on  two-dimensional 
computer  aided  design  and  drafting  (CAD-D)  tools,  and  much  of  it 
was  transferred  directly  from  the  DC-10  paper  drawings.  Three 
dimensional  wireframe  CAD-D  was  employed  on  about  40  percent  of 
the  configuration,  but  it  was  used  primarily  to  assist  in  making 
the  two-dimensional  drawings.  To  control  changes  to  the  design 
configuration  and  interfaces,  Douglas  had  an  administrative 
system  of  review,  approval,  and  implementation. 

Despite  residing  in  a  computer  system,  the  functional  drawings 
were  not  integrated.  With  separate  mechanical,  electrical, 
structural,  and  fuel  drawings  for  the  same  area,  and  different 
sets  of  people  responsible  for  these  different  drawings,  and 
these  different  sets  of  people  in  most  cases  reporting  to 
different  managers,  the  act  of  effectively  assessing  the  impacts 
of  changes  was  difficult.  As  a  result  of  this  arrangement  and  the 
integrated  avionics  system  interfacing  with  most  of  the 
aircraft's  subsystems,  there  were  many  interface  problems. 

[Larson,  1995] 

Despite  the  shortcomings  of  the  tools  and  methods  for  identifying 
interface  problems  and  controlling  changes,  such  issues  were 
eventually  resolved.  To  assist  in  the  process,  Douglas  developed 
an  order  of  precedence  for  interface  dispute  resolution.  The 
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areas  of  most  importance  to  least  importance  on  a  priority  scale 
were  (1)  pilot  control,  (2)  fuel  system,  (3)  environmental  ducts, 
(4)  electrical  system,  and  (5)  interiors/insulation  blankets. 
[Larson,  1995] 

The  interdisciplinary  nature  of  a  complex  system  like  the  MD-11 
suggests  an  early  design  involvement  by  a  variety  of  specialties 
along  with  engineering.  Perhaps  foremost  of  these  specialties  is 
manufacturing.  However,  early  manufacturing  involvement  did  not 
occur  to  the  extent  it  should  have.  As  a  redesign  effort,  Douglas 
did  not  intend  to  significantly  change  manufacturing  processes 
for  the  MD-11.  Furthermore,  as  mentioned  earlier,  a  majority  of 
the  MD-11  manufacturing  tooling  and  processes  were  taken  directly 
from  the  DC-10  production  line  with  the  assumption  that  this 
would  keep  the  up  front  program  costs  low.  Therefore,  there  were 
no  pressures  placed  on  engineering  and  manufacturing  personnel  to 
redesign  parts  for  the  purpose  of  lowering  production  cost. 

As  the  MD-11  weight  problems  and  cost  overrun  became  apparent  in 
the  middle  of  development,  Douglas  began  to  implement 
producibility  efforts  involving  manufacturing  that  have  been 
continued  into  production.  These  efforts,  called  design  for 
manufacturing  and  assembly  (DFMA) ,  were  focused  on  weight 
reduction,  production  cost  reductions  through  the  simplification 
of  parts,  and  production  cost  reductions  through  the  change  of 
assembly  order.  This  push  to  reduce  manufacturing  costs  was  also 
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driven  by  competitive  pressures  to  reduce  the  sales  price  of  the 
MD-11.  [Larson,  1995] 

During  MD-11  development,  there  was  no  systems  engineering 
management  plan  or  equivalent,  nor  has  there  been  a  separate 
systems  engineering  group.  In  the  original  organization,  the 
engineering  design  department  was  responsible  for  systems 
engineering.  The  MD— 11  chief  design  engineer,  who  was  also  the 
program  manager,  was  head  of  this  department.  He  worked  with  the 
heads  of  the  functional  engineering  specialties  in  the 
engineering  organization  under  him,  and  he  had  a  staff  of 
deputies  that  he  individually  assigned  to  work  technical 
problems.  The  deputies  were  responsible  for  identifying  and 
bringing  issues  to  the  chief  design  engineer  for  resolution. 
However,  these  issue  oriented  engineers  did  not  have  formal 
authority  over  the  non-engineering  technical  specialists,  the 
subcontractors,  or  even  some  of  the  engineers.  Furthermore,  there 
was  a  tendency  on  the  part  of  the  deputies  of  not  wanting  to 
escalate  issues  since  they  wanted  to  try  and  resolve  them 
themselves.  [Larson,  1995]  This  behavior  prevented  issues  from 
being  communicated  adequately  throughout  the  team  and  resolved  in 
a  more  timely  manner. 

In  addition  to  the  deputy  design  engineers,  an  independent  team, 
called  the  system  compliance  engineering  group,  was  chartered  to 
roam  throughout  the  program,  find  issues,  and  help  fix  them.  The 
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problem  with  this  overall  arrangement  was  that  nobody  was  clearly 
in  charge  of  interdisciplinary  issues  except  the  chief  design 
engineer.  The  lower-level  responsibilities  were  separated 
primarily  by  functional  specialty,  and  nobody  was  responsible  for 
the  cross-functional  activities  of  a  specific  area  of  the 
aircraft.  This  situation  contributed  to  the  interface 
difficulties  of  the  program. 

Major  subcontractors  played  critical,  and  sometime  leading,  roles 
in  MD-11  development.  While  the  key  subcontractors  had  on-site 
representatives  at  Douglas,  their  technical  work  was  done  back  at 
their  facilities.  Furthermore,  since  Douglas  had  no  permanent, 
on-site  technical  personnel  at  the  subcontractors'  locations, 
they  were  on  the  road  a  considerable  amount  of  time  visiting  the 
subcontractor  plants. 

The  delegation  of  critical  responsibilities  to  subcontractors 
freed  Douglas  of  some  work,  but  resulted  in  difficulties  in 
addition  to  large  amounts  of  travel.  Since  too  many  details  were 
vague  or  not  specified  in  the  specifications  and  subcontracts, 
Douglas  had  only  limited  control  over  what  activities  it  could 
get  the  subcontractors  to  perform.  This  became  more  of  an  issue 
as  the  design  changed  and  the  schedule  slipped  due  to  the 
underestimation  of  the  development  effort  by  Douglas.  While 
relations  between  Douglas  and  its  subcontractors  can  be 
considered  normal  overall,  some  scope  of  work  conflicts  did 
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result  in  legal  action  and  delays.  Because  of  this,  a  Douglas 
engineering  manager  believes  Douglas  should  have  done  more  in- 
house  integration.  [Larson,  1995]  Despite  the  difficulties, 
Douglas  eventually  succeeded  in  integrating  the  subsystems  and 
the  airframe. 

Recognizing  the  problems  inherent  in  the  functionally  oriented 
way  the  entire  corporation  did  business,  McDonnell  Douglas  tried 
to  implement  the  TQMS  mentioned  earlier.  The  MD-11  organization 
was  part  of  this  transformation,  and  it  occurred  in  the  middle  of 
the  aircraft's  development.  This  attempt  was  painful  to  the 
company  due  to  its  poor  execution.  The  change  took  much  longer  to 
implement  than  planned,  and  the  disruption  contributed  to  the 
slip  in  schedule.  The  MD-11  effort,  however,  did  eventually  move 
into  a  more  product  oriented  structure  with  interdisciplinary 
product  team  members  physically  located  together.  By  the  time 
this  structure  was  fully  implemented  and  functioning,  development 
was  complete. 

According  to  a  Douglas  engineering  manager,  a  crossfunctional 
teaming  arrangement  like  integrated  product  teams  would  have 
helped  MD-11  development  if  it  had  been  in  place  at  the  beginning 
of  the  program.  [Larson,  1995]  Such  a  structure  would  have 
improved  the  efficiency  of  technical  management  and  systems 
integration. 
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Despite  technical  management  problems,  life  cycle  issues  were 
addressed  by  the  MD-11  development  team  throughout  development. 
These  issues  included  reliability,  maintainability,  and  training. 
Such  supportability  issues,  though,  were  not  a  major  challenge. 
The  aircraft  was  designed  to  be  fully  compatible  with  existing 
ground  support  equipment  and  to  require  about  the  same  level  of 
maintenance  as  the  DC-10.  [Lenorovitz,  1991]  Training  was  also 
very  similar.  Additionally,  the  detailed  approach  to  achieve  the 
DC-10-level  safety  related  reliability  goals  was  contained  in  a 
reliability  plan,  and  it  was  verified  by  flight  testing. 
Therefore,  operating  costs  were  projected  to  be  as  good  if  not 
better  than  the  DC-10. 

Life  cycle  issues,  like  all  other  MD-11  issues,  were  ultimately 
the  responsibility  of  the  program  manager.  As  mentioned  earlier, 
the  program  manager  also  filled  the  role  of  chief  design 
engineer,  and  he  was  given  full  responsibility  to  execute  the 
program.  He  possessed  full  authority  over  engineering,  but  he  had 
to  appeal  to  a  vice-president  if  he  could  not  resolve  issues  with 
his  manufacturing  counterpart.  The  program  manager  set  the 
program  milestones  and  assessed  design  and  program  status  at 
periodic  technical  and  management  reviews.  Development  risk 
assessment  was  also  under  his  purview,  but  it  was  not  effectively 
carried  out  in  the  beginning  of  development. 

Helping  him  keep  track  of  the  effort  was  the  master  schedules 
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group,  which  had  the  capability  to  assess  schedule  impacts  to  the 
overall  program  when  problems  appeared.  The  program  manager's 
activities  were  also  supported  by  a  computerized  cost  accounting 
system  that,  although  not  very  timely,  provided  an  adequate  level 
of  visibility  into  cost  performance.  Such  tools  were  critical  to 
providing  control  in  a  less  than  ideal  development  environment. 

Development  Environment 

The  MD-11  was  developed  with  the  customer  in  mind,  but  direct 
interaction  between  Douglas  designers  and  potential  airline 
customers  was  minimal .  Marketing  surveys  were  the  key  means  of 
obtaining  information  on  top-level  airline  requirements  and 
desires  throughout  the  first  phases  of  development.  Although 
airlines  that  had  placed  firm  orders  were  able  to  order  custom 
features,  most  of  the  MD-11 's  design  details  were  left  to  the 
Douglas  engineers  to  determine  without  customer  review  or 
comment.  The  one  exception  was  the  MD-11  flight  deck  design  which 
evolved  directly  from  a  collaboration  between  Douglas  and  pilots 
from  potential  customer  airlines. 

If  airlines  had  been  invited  to  review  the  MD-11  design  at 
periodic  design  and  program  reviews,  they  would  have  seen  the 
detailed  technical  requirements  and  design  change  considerably 
during  development  and  production.  Only  a  few  of  these 
alterations  originated  with  the  customers  since  the  airlines  had 
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not  changed  their  minds  as  to  what  they  fundamentally  wanted  from 
the  MD-11.  The  changes  were  the  result  of  the  aircraft  not  being 
well  specified  up  front  by  Douglas.  Furthermore,  due  to  the  need 
to  alleviate  performance  shortfalls,  Douglas  was  continually 
introducing  design  fixes  during  development  and  production. 

Program  funding  and  workforce-level,  while  not  terribly  volatile, 
were  not  completely  stable,  either.  During  concept  development, 
Douglas  started  hiring  many  people  to  prepare  for  FSED  and 
production.  However,  once  the  program  ramped  up  at  full  speed, 
the  program  was  placed  on  hold  as  management  tried  to  decide  on 
the  design  configuration  and  whether  the  program  should  go  ahead 
or  not.  Also,  even  though  McDonnell  Douglas  had  approved  the 
three  year  budget  of  just  over  $500  million  at  the  beginning  of 
the  program,  the  effort  faced  reduced  funding  during  portions  of 
FSED  due  to  cash  flow  problems  caused  by  the  overrun.  Both 
situations  resulted  in  the  temporary  reduction  of  manpower. 

Despite  the  funding  reductions,  the  MD-11  received  support  from 
corporate  management  throughout  development.  The  MD— 11  program 
was  initially  sold  to  McDonnell  Douglas  corporate  management  as  a 
low— risk  derivative  of  the  DC— 10  that  would  only  reguire  small 
changes,  resulting  in  a  relatively  short  design  and  development 
cycle.  Since  McDonnell  Douglas  did  not  want  to  invest  much  of  its 
capital  in  commercial  aircraft,  preferring  the  lower  risk  of 
government  funded  cost-plus  development  programs,  approval  was 
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predicated  on  the  price  being  relatively  low.  The  program  cost 
estimate  that  Douglas  gave  to  corporate  management  was  low 
because  it  was  based  on  assumptions  of  small  changes  that  later 
turned  out  to  be  wrong.  [Larson,  1995]  While  McDonnell  Douglas 
was  not  pleased  with  the  schedule  slip  and  cost  overrun,  the 
program  was  still  supported  due  to  the  projected  profitability  of 
the  product  line. 

Along  with  corporate  support,  continuity  of  the  key  development 
team  members  was  maintained  to  a  moderately  high  degree.  Despite 
the  functional  structure,  specific  engineers  were  assigned  to  the 
effort  on  a  long  term  basis.  Furthermore,  many  key  designers  and 
managers  during  concept  development  remained  on  the  program  when 
it  entered  FSED  and  production.  The  continuity  of  the  development 
team  and  the  stability  of  the  organizational  structure,  though, 
were  seriously  impacted  temporarily  by  the  1989  TQMS 
implementation . 

During  this  transformation,  the  MD-11  program  completely 
reorganized.  Positions  changed  and  everybody  had  to  reapply  for 
new  positions.  [Larson,  1995]  Despite  the  chaos,  most  of  the  key 
designers  and  managers  who  had  been  on  the  program  from  the 
beginning  eventually  made  it  back  under  the  new,  integrated 
product  teaming  structure. 
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Douglas  employees  covered  a  broad  range  of  expertise  and 
background,  and  they  did  not  always  cooperate  as  well  as  they 
could  have.  This  was  true  of  the  relationship  between  the  Douglas 
engineering  and  manufacturing  organizations.  Douglas  is  a 
production  oriented  company  in  which  the  greatest  amount  of  power 
is  resident  in  the  manufacturing  organization.  Like  many  other 
companies,  Douglas  had  a  cultural  wall  between  the  two 
disciplines.  In  the  Douglas  culture,  manufacturing  is  never  happy 
with  what  the  engineering  group  gives  them  and  viewed  design 
engineering  negatively  as  an  overhead  function  that  did  not 
generate  company  revenues.  [Larson,  1995]  Even  though  the 
working-level  relationships  between  engineering  and  manufacturing 
were  positive,  the  company  atmosphere  did  not  promote  closer 
working  relationships.  The  implementation  of  the  TQMS  was  partly 
meant  to  remove  the  barriers  between  them,  but  did  not  have 
immediate  success. 

Concerning  the  relationship  between  Douglas  and  the 
subcontractors,  they  can  generally  be  described  as  businesslike, 
but  not  particularly  close.  As  mentioned  previously,  poor 
specification  development  by  Douglas  and  technical  problems 
increased  the  amount  and  the  scope  of  effort  required  from  some 
of  the  subcontractors,  resulting  in  conflicts  involving  legal 
actions.  Although  the  TQMS  was  partly  aimed  at  increasing 
cooperation  between  Douglas  and  its  subcontractors,  those 
relationships,  at  least  initially,  were  not  significantly 
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impacted  positively  or  negatively  by  the  change  in  organizational 
structure  and  operating  philosophy,  except  it  did  create 
confusion  as  the  Douglas  points  of  contact  continually  changed. 

Relations  between  Douglas  and  the  airlines  were  good  at  the 
beginning,  but  again,  not  particularly  close,  since  interaction 
with  the  customer  was  limited.  When  problems  with  the  initial  MD- 
11s  surfaced,  the  relations  with  some  MD-11  recipients  turned 
negative.  The  large  effort  by  Douglas  to  solve  the  problems  was 
crucial  to  regaining  the  confidence  of  the  customer  airlines. 

The  means  of  interaction  between  all  members  of  the  MD-11 
development  team  were  primarily  telephones,  face-to-face 
meetings,  and  mail.  While  adeguate,  it  was  not  an  efficient 
arrangement  for  transferring  large  amounts  of  technical 
information.  Also,  meetings  to  coordinate  activities  and  resolve 
problems  with  the  subcontractors  were  expensive  and  required 
greater  time  to  allow  for  travel  since  the  MD-11  participants 
were  widely  distributed  geographically.  While  most  Douglas 
workers  were  located  within  several  miles  of  each  other  in  Long 
Beach,  CA. ,  everyone  else  was  dispersed.  Honeywell  is  in  Phoenix, 
AZ.,  Pratt  &  Whitney  and  General  Electric  are  on  the  East  Coast, 
and  the  many  other  subcontractors  and  vendors  were  located 
throughout  the  United  States  and  even  the  world.  To  facilitate 
communication,  the  key  subcontractors  had  on— site  representatives 
at  Douglas  facilities,  and  Douglas  personnel  spent  a  large  amount 
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of  time  at  the  major  subcontractor  plants. 

Douglas  is  a  big  company  in  a  large  corporation,  and  it  had  its 
share  of  bureaucratic  procedures  and  approval  chains  to  comply 
with  during  MD— 11  development,  as  most  other  systems  contractors. 
Furthermore,  Douglas's  configuration  change  control  activities 
were  not  structured  for  quick  and  efficient  changes.  Despite 
these  difficulties,  Douglas  did  have  the  flexibility  to  tailor 
business  relationships  without  significant  government  involvement 
as  is  the  case  in  government  sponsored  projects.  Workers  followed 
the  established  procedures  of  Douglas  Aircraft  Company,  and  the 
top-level  managers  were  given  the  responsibility  and  authority  to 
carry  out  the  development  effort  without  micromanagement  from  the 
corporation.  Not  until  the  schedule  slip  and  overrun  started  to 
materialize  did  corporate  managers  become  more  involved.  This 
involvement  was  not  all  negative,  since  technical  experts  from 
outside  of  Douglas  were  provided  to  assist  in  solving  the 
problems  causing  the  delay.  [Larson,  1995] 

As  for  the  numerous  subcontractors  and  suppliers,  they  were 
managed  in  a  traditional  manner .  That  is ,  Douglas  maintained 
oversight  through  on— site  visits  and  correspondence,  and  the 
subcontractors  were  given  the  flexibility  to  do  what  they  needed 
to  do  without  burdensome  procedures  to  follow. 


As  one  of  the  biggest  producers  of  large  commercial  jet  aircraft 
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in  the  world,  Douglas  had  a  considerable  amount  of  manufacturing 
expertise  and  experience  inherent  in  its  4,000  to  6,000  workforce 
during  the  height  of  MD-11  development  and  production.  At  the 
beginning  of  the  MD-11  effort,  the  company  was  solely  a 
production  house  producing  the  MD— 80  and  DC— 10,  and  it  had  not 
developed  a  new  aircraft  in  years.  Therefore,  it  did  not  possess 
a  large  pool  of  design  engineering  and  development  expertise. 
[Larson,  1995]  As  a  consequence,  a  large  number  of  engineers 
were  hired  to  conduct  the  program.  Although  the  engineering 
design  team  was  technically  capable,  they  of  course  lacked  the 
experience  of  working  together  on  a  development  program. 

Given  that  the  amount  of  profit  obtained  from  the  MD-11  is  due 
significantly  to  the  number  of  aircraft  sold,  given  that  the 
number  of  sales  is  tied  to  how  well  the  MD— 11  operates  in  service 
and  satisfies  the  customers,  and  given  that  the  future  Douglas 
aircraft  will  be  judged  partly  on  the  reputation  of  the  MD-11, 
the  company  had  a  lot  of  incentive  to  ensure  the  MD-11  performed 
to  customer  expectations.  The  basic  warranty  offered  to  all  MD-11 
customers  was  supposed  to  ensure  this .  In  addition  to  workmanship 
and  parts  quality  provisions,  Douglas  guaranteed  payload/range 
performance  that  would  enable  certain  non-stop,  international 
flights.  While  performance  was  not  attained  by  the  initial  MD-lls 
delivered,  Douglas  developed  modifications  and  retrofit  the 
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Summary /Conclusion 

The  MD-11,  intended  as  a  low-risk  redesign,  turned  out  to  be  a 
complex  systems  integration  effort.  The  development  team  failed 
to  anticipate  the  integration  challenges,  and  it  was  not 
structured  to  handle  them  efficiently.  The  nine  month  delivery 
delay  of  an  original  three  year  schedule  and  a  30  percent  overrun 
are  significant  for  a  redesign  that  did  not  require  breakthrough 
technology.  Despite  the  problems  with  the  development  process, 
the  eventual  attainment  of  performance  requirements,  the 
potential  for  further  improvements,  and  the  number  of  orders 
ensuring  program  profitability  to  McDonnell  Douglas  suggest  the 
MD-11  is  a  reasonably  successful  design. 
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Table  3.6-1  MD-11  Performance  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

TECHNICAL  PERFORMANCE  - 
INITIAL 

0-10 

1 

3 

3 

TECHNICAL  PERFORMANCE  - 
MATURE 

o 

1 

h-1 

o 

1 

10 

10 

COST  PERFORMANCE 

0-10 

1 

5 

5 

SCHEDULE  PERFORMANCE 

0-10 

1 

4 

4 

PERFORMANCE  TOTAL 

■  1 1 

22 

Table  3.6-2  MD-11  Systems  Engineering  Fundamentals  scores. 
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Table  3.6-3  MD-11  Development  Environment  scores. 


Figures  of  Merit 

Range 

Weight 

Rating 

Score 

EMPHASIS  ON  THE  CUSTOMER 

0-10 

1 

6 

6 

STABILITY  OF  REQUIREMENTS 

AND  CONFIGURATION 

0-10 

1 

4 

4 

FUNDING  AND  WORKFORCE-LEVEL 
STABILITY 

0-10 

1 

6 

6 

STRONG  SUPPORT  FOR  PROGRAM 

0-10 

1 

6 

6 

CONTINUITY  OF  CORE 

DEVELOPMENT  TEAM 

0-10 

1 

5 

5 

STABILITY  OF  ORGANIZATIONAL 
STRUCTURE 

0-10 

1 

2 

2 

COOPERATION  AMONG  ALL 
STAKEHOLDERS 

0-10 

1 

5 

5 

EFFECTIVE  COMMUNICATION 
WITHIN  TEAM 

0-10 

1 

5 

5 

FLEXIBILITY  AND  AUTONOMY 

0-10 

1 

6 

6 

WORKFORCE  EXPERTISE  AND 
EXPERIENCE 

0-10 

1 

5 

5 

ACCOUNTABILITY  FOR  SYSTEM 
PERFORMANCE 

0-10 

1 

8 

8 

DEVELOPMENT  ENVIRONMENT 

IMS 

58 
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Table  3.6-4  MD-11  Design  Difficulty  scores. 


Elements 

Range 

Score 

TYPE 

0-15 

4 

KNOWLEDGE  COMPLEXITY 

0-10 

5 

STEPS 

0-10 

9 

QUALITY  IMPLEMENTATION 

EFFORT 

o 

1 

h- 1 
O 

6 

MANUFACTURING  OPERATIONS 
IMPLEMENTATION  EFFORT 

0-5 

4 

SELLING  PRICE  CONSTRAINT 

0-5 

4 

DESIGN  DIFFICULTY  TOTAL 

0-55 

32 

Table  3.6-5  MD-11  Resources  scores. 


Elements 

Range 

Score 

COST 

0-15 

10 

TIME 

0-10 

6 

INFRASTRUCTURE 

0-10 

7 

RESOURCES  TOTAL 

0-35 

23 
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CHAPTER  4 

RESULTS  COMPARISON,  ANALYSIS,  AND  DISCUSSION 


The  results  of  the  case  study  ratings  are  summarized  in  Tables 
4.1  through  4.5.  The  total  scores  were  used  to  explore  possible 
relationships  between  the  characteristics  of  the  system 
development  process.  Due  to  the  limited  number  of  data  points,  no 
strong  conclusion  can  be  made.  However,  as  discussed  later,  there 
appears  to  be  a  generally  positive  relationship  between  the 
performance  scores  and  the  systems  engineering  fundamentals 
scores  for  the  aircraft  development  case  studies.  Before 
exploring  this  issue,  the  cases  are  discussed  in  relation  to  one 
another. 

The  six  case  studies  were  presented  in  the  order  of  higher  to 
lower  systems  engineering  fundamentals  score,  as  indicated  in 
Table  4.2.  However,  all  six  aircraft  are  considered  successful 
systems  in  that  they  all  operate  to  a  high  degree  of  ability,  as 
indicated  by  the  technical  performance-mature  figure  of  merit  in 
Table  4.1.  Combining  technical  performance  with  the  cost  and 
schedule  performance  figures  of  merit  provides  an  effectiveness 
index  of  the  overall  system  development  process  itself.  The  777 
development  was  by  far  the  highest  rated  effort,  and  it  was 
followed  in  order  by  the  F-117,  B-2,  C-17,  Model  60,  and  MD-11. 
Tables  4.2  and  4.3  clearly  point  out  their  main  strengths  and 
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DEVELOPMENT  ENVIRONMENT  TOTAL 


Table  4.4  Summary  of  Design  Difficulty  scores. 
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weaknesses  in  terms  of  the  systems  engineering  fundamentals  and 
development  environment . 

The  Boeing  777  development  has  shown  itself  to  be  successful  by 
its  on-time  production  and  delivery  of  an  aircraft  meeting 
customer  requirements.  The  777  received  the  highest  ratings  of 
all  six  cases  in  performance  (36)  and  systems  engineering 
fundamentals  (126),  and  second  highest  in  development  environment 
(96).  Boeing  effectively  utilized  the  full  range  of  systems 
engineering  principles,  tasks,  and  techniques  through 
implementation  of  concurrent  engineering.  Boeing  also  created  an 
environment  that  strongly  supported  the  work  and  communication  of 
the  interdisciplinary  design/build  teams.  The  success  is  due 
specifically  to  thorough  requirements  generation,  the  close 
participation  of  the  airline  customers  throughout  the  entire 
development,  the  advanced  computer  aided  design  and  manufacturing 
system,  and  the  longer  planned  full-scale  engineering  development 
(FSED)  schedule  that  ensured  development  problems  were  worked  out 
before  the  first  customer  delivery.  The  result  was  a  development 
with  no  apparent  areas  of  significant  weakness. 

The  next  two  aircraft,  the  F-117  and  the  B-2,  are  both 
revolutionary  aircraft  of  high  design  difficulty  and  technical 
risk  that  followed  strong  systems  engineering  practices  to  help 
develop  the  highly  integrated  designs.  Both  aircraft  possessed 
strong  technical  performance  from  their  introductions,  and  they 
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have  satisfied  their  Air  Force  customer.  Schedule  and  cost 
performances,  however,  were  not  laudatory,  especially  for 
Northrop.  The  uncertainties  surrounding  the  development  of 
complex,  breakthrough  technologies  certainly  contributed  to  cost 
growth  and  delays,  and  the  highly  integrated  nature  of  both 
designs  generated  many  new  issues  to  resolve.  This  is  true 
despite  the  fact  that  both  programs  carried  out  extensive  risk 
reduction  activities  prior  to  entering  FSED.  Furthermore,  as 
classified  military  efforts,  both  were  developed  by  highly 
skilled  and  motivated  workers  in  highly  stable  environments  that 
maintained  strong  funding  and  political  support  throughout  most 
of  their  respective  FSED  phases.  Further  countering  the  favorable 
environment  for  the  B-2  was  a  major  requirements  change  from  the 
Air  Force  early  in  FSED,  resulting  in  a  costly  redesign  and  an 
extension  in  schedule.  The  momentum  of  the  B-2  program  was  also 
affected  by  the  Air  Force's  delay  in  starting  FSED  to  accommodate 
the  resurrected  B-l  program.  The  F-117  did  not  experience  a  major 
change  in  customer  requirements  like  the  B-2,  but  its  FSED 
schedule  and  budget  were  impacted  by  the  crash  of  the  first 
production  aircraft  due  to  an  assembly  error  not  detected  during 
ground  inspection  and  testing. 

Despite  both  efforts  strongly  following  systems  engineering 
principles,  the  aircraft  were  developed  differently.  While  the 
smaller  F-117  was  designed  and  fabricated  by  a  relatively  small 
project  team  in  accordance  with  the  philosophy  of  the  Lockheed 
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Skunk  Works,  Northrop  developed  a  new  approach  to  develop  the 
large  B-2.  Interestingly,  most  of  Northrop 's  approach  was  later 
adopted  by  Boeing,  a  B-2  subcontractor,  for  its  development  of 
the  777. 

Both  the  B-2  and  777  centered  their  efforts  around  a  digital 
"paperless"  database,  set  up  extensive  in-house  integration 
testing  laboratories,  utilized  an  avionics  test  bed  using  other 
aircraft,  and  planned  for  extensive  flight  testing  beyond  what 
was  required  for  certification.  One  major  difference  was  the  way 
the  efforts  were  organized.  Northrop  had  a  functionally  oriented 
team  that  used  an  informal  zone  management  structure  to  deal  with 
interdisciplinary  and  interface  issues,  whereas  Boeing  developed 
formal  product  teams.  Furthermore,  Boeing's  computer  aided  design 
and  manufacturing  system  was  also  more  advanced.  It  was  able  to 
perform  virtual  prototyping,  thereby  identifying  interface 
problems  automatically  on  computer.  Therefore,  Boeing  essentially 
refined  and  successfully  implemented  the  development  approach 
that  Northrop  had  pioneered.  The  B-2  and  777  examples  are 
significant  in  that  they  indicate  the  direction  all  aircraft 
development  is  moving  to  deal  with  increasing  degrees  of  aircraft 
complexity  and  integration. 


Of  comparable  design  difficulty  to  the  111  was  the  C  17  military 
transport.  However,  its  performance  score  of  23  is  significantly 
lower  in  comparison.  Despite  a  high  mature  technical  performance 
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score,  the  initial  technical  performance  was  not  acceptable,  and 
cost  and  schedule  performances  have  been  marginal.  While  the 
early  requirements  development  and  design  work  with  the  Air  Force 
were  commendable,  actions  by  Douglas  throughout  FSED  as  well  as 
imposed  conditions  have  resulted  in  great  difficulties.  Perhaps 
as  its  first  mistake,  Douglas  contributed  to  its  many  eventual 
problems  by  accepting  more  stringent  performance  requirements 
than  what  the  customer  had  originally  requested. 

Much  of  the  performance  characteristic  shortfalls,  however,  have 
been  due  to  poor  management  on  the  contractor's  part.  Douglas's 
configuration  and  production  management  systems  were  inadequate, 
and  its  program  management  function  was  ineffective.  These 
problems  enhanced  as  well  as  were  enhanced  by  a  very  volatile 
environment,  much  of  it  defined  by  numerous  design  changes,  high 
rates  of  repair  and  rework,  the  massive  Total  Quality  Management 
System  ( TQMS )  reorganization,  opposition  from  Congress, 
acrimonious  relations  with  the  customer ,  and  continuous  turnover 
of  the  workforce.  Further  contributing  to  some  of  the  instability 
was  the  long,  drawn-out  schedule  with  delayed  FSED  start.  The 
C-17  development  evidences  many  of  the  same  problems  as  the  MD- 
11,  which  was  developed  by  the  commercial  sector  of  the  same 
contractor  during  the  same  time. 

The  Learjet  Model  60  was  rated  with  a  high  technical  performance 
score.  However,  like  most  of  the  other  aircraft  presented,  the 
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overall  performance  score  was  tempered  somewhat  by  low  schedule 
and  cost  performance  scores.  For  the  relatively  uncomplicated 
redesign  effort  by  a  small  development  team  that  carried  out  many 
of  the  fundamental  systems  engineering  practices  to  a  reasonable 
degree,  this  is  unfortunate.  Lear jet's  major  shortcoming  with  the 
Model  60  effort  was  poor  requirements  development  stemming  from 
the  lack  of  communication  and  coordination  between  the  sales 
department  and  the  engineering  group.  This  situation  resulted  in 
a  redirection  after  the  initial  design  had  been  completed.  With 
the  exception  of  the  changing  requirements  and  internal 
communication  problems,  the  small  team  development  environment 
was  quite  conducive  to  systems  engineering  activities. 

Like  the  Model  60,  the  MD-11  was  a  stretched  version  of  an 
existing  airframe  with  improved  integrated  avionics,  engines,  and 
aerodynamics.  Also  like  the  Model  60,  one  of  the  MD-11 's  biggest 
problems  dealt  with  requirements  development.  The  failure  to 
develop  a  single,  distinct  specification  for  the  MD-11  before 
FSED  and  the  attendant  underestimation  of  the  incipient  system 
design  effort  helped  lead  to  the  unanticipated  systems  and 
interface  problems.  Such  problems  were  exacerbated  by  marginally 
effective  mechanisms  for  configuration  and  production  management 
as  well  as  a  development  environment  not  highly  supportive  of 
strong  systems  engineering  activities.  The  situation  worsened 
during  the  poorly  executed  McDonnell  Douglas  TQMS  reorganization, 
thereby  greatly  impacting  progress  on  the  MD-11  effort  towards 
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the  end  of  FSED.  Consequently,  the  performance  score  reflects 
only  a  moderate  overall  success,  having  been  downgraded  by  the 
schedule  delays,  cost  growth,  and  problems  with  initially 
delivered  aircraft. 

The  picture  that  emerges  most  prominently  from  the  aircraft  case 
studies  is  that  poor  requirements  development,  inadequate 
configuration  management,  and  weak  program  and  technical 
management  appear  to  be  the  primary  determinants  of  lower  systems 
engineering  fundamentals  scores.  Significant  development 
environment  instability  is  reflected  in  the  scores  of  several  of 
the  cases  as  well,  primarily  in  the  categories  of  stability  of 
requirements  and  configuration,  funding  and  workforce-level 
stability,  strong  support  of  program,  continuity  of  core  team 
members,  stability  of  organizational  structure,  and  cooperation 
of  all  stakeholders.  Whether  or  not  these  areas  are  the  primary 
detriments  to  development  environment  scores  for  other  categories 
of  systems  is  not  known. 

Comparing  the  systems  engineering  fundamentals  scores  for  the 
cases,  the  military  programs  appear  to  tend  towards  higher 
scores.  This  is  probably  due  to  the  fact  that  the  government 
usually  mandates  a  high  degree  of  systems  engineering  on  its 
large  system  development  efforts.  However,  a  commercial  program 
received  the  highest  systems  engineering  fundamentals  score. 


An  important  question  to  consider  now  is  whether  or  not  the  case 
study  ratings  suggest  anything  about  a  possible  relationship 
between  systems  engineering  fundamentals  and  performance.  To 
explore  this  issue,  a  simple  graphical  analysis  was  used. 

Figure  4.1  is  a  plot  of  the  systems  engineering  fundamentals  (SE) 
scores  versus  the  performance  scores  (Performance) .  The  plot 
illustrates  a  generally  positive  relationship  between  the  two 
characteristics.  If  the  scoring  methodology  is  valid,  then  this 
would  indicate  that  the  higher  the  degree  of  systems  engineering 
performed,  the  higher  the  success  of  a  system  development  effort. 

Figure  4.1  Systems  Engineering  Fundamentals  (SE)  vs.  Performance 
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The  commercial  aircraft  appear  to  follow  the  pattern  better  than 
the  military  aircraft.  The  777,  Model  60,  and  MD-11  fall  nearly 
in  a  straight  line,  while  the  three  military  aircraft  fall  below 
it  and  appear  to  be  aligned  with  a  smaller  slope.  The  points  that 
are  farthest  from  the  trend  line  are  the  F-117  and  the  B-2.  The 
performance  scores  for  these  two  breakthrough  designs  may  be 
adversely  affected  by  the  significant  design  difficulty  and 
technical  risk  associated  with  each.  If  this  is  accurate,  then 
design  difficulty  could  possibly  be  used  in  a  methodology  to 
generate  a  multiplicative  factor  to  adjust  the  performance  scores 
closer  to  the  commercial  aircraft  trend  line. 

The  C-17  is  also  below  the  trend  line,  but  its  design  difficulty 
is  not  as  high  as  the  stealth  aircraft.  However,  the  deviation 
may  be  attributed  to  the  program's  volatile  environment  as 
reflected  by  its  low  development  environment  score.  This  also 
suggests  that  perhaps  a  multiplicative  factor  methodology  may  be 
developed  based  on  development  environment  scores  to  help  bring 
the  plots  of  all  the  efforts  in  line. 

As  just  discussed,  the  plot  locations  of  the  government  programs 
in  Figure  4.1  can  probably  be  explained  by  both  the  differences 
in  design  difficulty  and  development  environment .  However, 
another  possible  way  of  looking  at  the  discrepancies  may  be 
simply  in  terms  of  government  versus  commercial .  Perhaps  a  single 
multiplicative  factor  based  only  on  whether  or  not  the 
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development  was  a  government  effort  can  be  developed.  Since  all 
three  military  efforts  had  generally  marginal  cost  and  schedule 
performance,  this  suggests  that  the  development  process  for 
government  aircraft  is  inherently  more  prone  to  cost  and  schedule 
difficulties.  By  multiplying  those  figures  of  merit  scores  by  a 
factor  of  two,  the  performance  numbers  are  enhanced.  The  modified 
numbers  are  presented  in  Figure  4.2. 

Figure  4.2  SE  vs.  Performance  (modified) 


Whether  such  a  factor,  or  other  potential  factors  mentioned,  is 
valid  is  mere  speculation  without  further  data  to  analyze. 
Therefore,  the  impact  of  design  difficulty ,  development 
environment,  and  government  procedures  in  the  system  development 
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process  is  an  area  requiring  further  study. 

One  characteristic  of  the  system  development  process  not 
discussed  yet  is  resources.  The  resources  scores  were  fairly 
comparable  for  all  the  aircraft,  as  shown  in  Table  4.5.  This  is 
reasonable,  in  that  all  large  and  highly  complex  aircraft  need 
most  of  the  same  types  of  components  and  subsystems  and  utilize 
much  of  the  same  supplier  infrastructure. 

The  results  of  plotting  design  difficulty  versus  resources,  as  in 
William  L.  Chapman's  Ph.D.  dissertation  referenced  in  Chapter  1, 
are  shown  in  Figure  4.3.  This  indicates  that  aircraft  generally 
follow  a  linear  relationship  with  regard  to  the  two 
characteristics.  Although  Chapman  concluded  that,  across  all 
system  designs,  the  two  characteristics  were  not  linearly 
related,  he  did  recognize  that  the  plotted  points  of  systems 
within  some  categories  are  grouped  together . 

While  far  from  being  significant  evidence,  the  graphs  do  provide 
at  least  a  hint  of  support  for  the  contentions  expressed  above, 
thereby  suggesting  that  the  methodology  and  approach  proposed  by 
this  thesis  may  have  some  merit.  However,  further  work  is  needed 
to  support  this  assertion. 

While  much  effort  went  into  the  development  of  the  rating 
methodology,  it  is  conceded  that  improvements  can  be  made.  Some 


Figure  4.3  Design  Difficulty  vs.  Resources 


ideas  that  may  enhance  the  process  are  provided  as  follows. 

The  figures  of  merit  for  systems  engineering  fundamentals  can  be 
considered  reasonably  complete  based  on  the  existing  systems 
engineering  literature.  However,  they  may  be  broken  out  in 
different  combinations  than  what  was  presented.  As  for  the 
development  environment  figures  of  merit,  they  could  be  refined 
in  some  instances  to  be  more  precise  and  measurable.  Perhaps  the 
most  important  issue  that  needs  further  attention  is  the  relative 
weighting  between  the  figures  of  merit  within  the  system 


251 


development  characteristics.  It  may  be  appropriate  to  make 
modifications  if  justification  can  be  found.  Review  of  the 
characteristics,  figures  of  merit,  and  rating  criteria,  and 
scoring  system  by  others  in  the  fields  of  systems  engineering, 
engineering  management,  organizational  effectiveness,  and 
decision  analysis  would  be  helpful  towards  making  improvements. 

In  addition  to  investigating  possible  refinements  to  the 
methodology,  additional  case  studies  are  required  to  provide  data 
to  support  or  refute  the  methodology  presented  and  the  assertions 
made  earlier.  Additional  aircraft  data  points  are  needed  to 
provide  a  statistically  significant  database  to  allow  a  more 
rigorous  analysis  to  be  conducted  to  test  whether  or  not  the 
results  presented  above  represent  more  than  coincidence. 
Furthermore,  cases  are  needed  from  other  than  aircraft  systems  to 
determine  if  the  suggested  positive  relationship  between  systems 
engineering  fundamentals  and  performance  hold  up  for  all 
categories  of  systems,  from  complex  consumer  electronics  to  large 
construction  projects.  Also,  any  further  investigation  should 
attempt  to  determine  how  design  difficulty  and  the  level  of 
resources  relate  to  the  outcomes  of  the  overall  system 
development  process.  Additional  work  will  help  determine  if  the 
proposed  methodology  provides  for  valid  "apples— to— apples" 
comparison  between  widely  different  categories  of  systems. 
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CHAPTER  5 

VALIDATION  OF  METHODOLOGY 


The  case  study  evaluation  and  scoring  methodology  presented  and 
demonstrated  in  this  thesis  was  subjected  to  a  validation  trial. 
Nine  engineers  consisting  of  eight  graduate  students  and  a 
professor  in  Systems  Engineering  at  the  University  of  Arizona 
read  versions  of  the  case  studies  and  rated  them  according  to  the 
methodology.  The  scores  were  compiled  for  each  characteristic  for 
each  case,  and  the  range,  average,  and  standard  deviation  (S.D.) 
of  each  were  used  in  comparing  them  to  the  ratings  given  by  the 
author . 

The  mean  and  variability  of  the  engineers'  scores  were  calculated 
as  a  way  of  assessing  how  clearly  the  cases  were  written  and  the 
rating  criteria  were  defined.  If  the  scores  for  a  particular 
figure  of  merit  were  widely  varied  for  all  of  the  case  studies, 
then  it  would  suggest  that  the  rating  criterion  would  need  to  be 
more  specific.  If  the  scores  for  a  particular  figure  of  merit  for 
a  particular  case  varied  widely,  it  would  suggest  that  a  portion 
of  that  case  would  need  to  be  rewritten  to  make  it  clearer.  If 
the  average  of  the  scores  were  significantly  different  from  the 
rating  assigned  by  the  author,  it  would  suggest  that  the  case, 
the  criteria,  or  both  would  need  to  be  modified.  Finally,  such 
differences  could  also  indicate  an  inaccuracy  or  error  with  the 
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author's  rating. 

The  results  are  presented  in  Table  5.1  through  Table  5.24.  They 
show  the  differences  between  and  similarities  with  the  nine 
engineers'  ratings  and  the  author's  ratings. 

In  a  few  instances,  figures  of  merit  and  elements  and  their 
corresponding  criteria  changed  after  the  validation  trial  for 
reasons  not  directly  related  to  the  trial.  Those  figures  of  merit 
and  elements  are  outlined  in  gray,  and  they  are  not  useful  data 
points  in  relation  to  assessing  the  final  methodology.  The  new 
figures  of  merit  and  elements  are  listed  with  some  of  their  boxes 
blackened  since  they  were  not  used  during  the  validation  run.  The 
rows  that  do  not  have  any  shading  represent  figures  of  merit  and 
elements  that  remain  with  possible  minor  modifications  from  the 
original  methodology. 

The  figures  of  merit  and  elements  with  average  scores  differing 
by  two  or  more  from  the  author's  scores  (accuracy  criterion)  and 
with  standard  deviations  of  two  or  greater  (precision  criterion) 
were  closely  scrutinized.  Tables  5.31  through  5.35  are  analyses 
of  the  results  presented  by  system  development  characteristic, 
and  they  indicate  whether  or  not  the  figures  of  merit  and 
elements  were  subject  to  reevaluation  due  to  violating  the 
accuracy  criterion,  the  precision  criterion,  or  both. 
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Since  the  analyses  did  not  identify  any  figures  of  merit  or 
elements  with  problems  across  all  the  case  studies,  the  author 
concentrated  on  modifying  the  individual  cases.  However,  make-or- 
buy  figure  of  merit  was  modified  to  make  it  clearer.  Although  the 
design  difficulty  element,  type,  appears  to  have  a  problem,  the 
author  chose  to  deal  with  the  score  variability  through  case 
study  improvement. 

A  wide  range  of  scores  is  evident  for  some  of  the  case  studies. 
This  reflects  differences  in  understanding  and  perception  among 
the  participants  and  is  to  be  expected  for  people  who  have  not 
been  exposed  to  the  methodology  before. 

As  mentioned  earlier,  several  figures  of  merit  and  elements  were 
changed  for  reasons  other  than  the  validation  results.  Technical 
performance  was  split  into  two  measures  to  enable  easier 
evaluation,  and  two  design  difficulty  elements,  quality  and 
quantity,  were  eliminated  and  replaced  by  quality  implementation 
effort,  manufacturing  operations  implementation  effort,  and 
selling  price  constraint.  They  were  deemed  to  be  more  significant 
and  appropriate  measures. 

While  the  validation  results  motivated  the  review  of  a  number  of 
figures  of  merit  and  elements  and  the  cases  themselves,  it  is 
interesting  to  note  that  the  average  of  each  characteristic  total 
score  (as  well  as  the  total  of  the  average)  from  the  validation 
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group  compared  favorably  with  the  author's  total  scores.  This  is 
true  especially  for  the  ratings  for  systems  engineering 
fundamentals.  This  suggests  that  on  average,  the  group  judgment 
will  produce  a  numerical  score  reasonably  close  to  that  presented 
by  the  author.  The  author  suggests  that  the  improvements  made  to 
the  case  studies  and  people's  greater  experience  with  the 
methodology  will  improve  the  accuracy  and  precision  of  the 
scores.  An  additional  validation  run  using  engineers  is 
recommended  to  test  this  assertion. 


Table  5.8  Validation  Data  -  F-117  Systems  Engineering  Fundamentals 


Table  5.9  Validation  Data  -  B-2  Systems  Engineering  Fundamentals 


Table  5.10  Validation  Data  -  C-17  Systems  Engineering  Fundamentals  scores 


Table  5.11  Validation  Data  -  Model  60  Systems  Engineering  Fundamentals  scores 


Table  5.12  Validation  Data  -  MD-11  Systems  Engineering  Fundamentals 


Table  5.13  Validation  Data  -  111  Development  Environment  scores 


DEVELOPMENT  ENVIRONMENT  102.55 

TOTAL  110  85  101.78 


Table  5.15  Validation  Data  -  B-2  Development  Environment  scores 
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Table  5.16  Validation  Data  -  C-17  Development  Environment  scores 


Table  5.17  Validation  Data  -  Model  60  Development  Environment  scores 
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DEVELOPMENT  ENVIRONMENT  TOTAL  85.00 

103  74  84.22 


Table  5.18  Validation  Data  -  MD-11  Development  Environment  scores 
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Table  5.19  Validation  Data  -  777  Design  Difficulty 
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Table  5.21  Validation  Data  —  F— 117  Design  Difficulty  scores 
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Table  5,29  Validation  Data  -  MD-11  Design  Difficulty  scores. 
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Table  5.31  Performance  Validation  Analysis 


Case  Study 

Figure  of  Merit 

Violates 

Accuracy 

Violates 

Precision 

F-117 

Schedule  perform. 

X 

X 

B-2 

Cost  performance 

X 

Model  60 

Cost  performance 

X 

Table  5.32  Systems  Engineering  Fundamentals  Validation  Analysis 


Case  Study 


F-117 


Model  60 


MD-11 


Figure  of  Merit 


Make-or-buy 


Manufacturing 

considerations 


Incipient  system 
design 


Evaluating  alt. 
concepts 


Make-or-buy 


Validation 


Manufacturing 

considerations 


Requirements 

development 


Evaluating  alt. 
concepts 


Make-or-buy 


Manufacturing 

considerations 


Validation 


Configuration 

management 


Life  cycle 
considerations 


Program  managment 


Violates 

Accuracy 


Violates 

Precision 
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Table  5.33  Development  Environment  Validation  Analysis 
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Table  5.34  Design  Difficulty  Validation  Analysis 


Case  Study 

Element 

Violates 

Accuracy 

Violates 

Precision 

111 

Type 

X 

X 

F-117 

Type 

X 

C-17 

Type 

X 

Model  60 

Steps 

X 

MD-11 

Type 

X 

Table  5.35  Resources  Validation  Analysis 
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CHAPTER  6 
SUMMARY 

This  thesis  has  proposed  a  universal  methodology  to  score  case 
studies  of  system  development  efforts  according  to  figures  of 
merit  and  element  rating  criteria.  Also  proposed  in  an  approach 
to  explore  the  relationships  between  the  characteristics  of  the 
system  development  process,  particularly  the  magnitude  of  the 
application  of  systems  engineering,  and  the  degree  to  which  that 
process  is  successful.  Chapter  1  laid  the  foundation  by  pointing 
out  that  the  lack  of  a  common  definition  of  systems  engineering 
makes  development  of  a  rating  system  difficult,  and  it  provided  a 
model  which  provides  the  framework  for  the  methodology.  It  also 
outlines  the  entire  approach  to  investigating  possible  high  level 
relationships . 

Chapter  2  presents  the  rating  criteria  and  scoring  method  for 
each  of  the  figures  of  merit  and  elements  for  the  five 
characteristics  of  the  system  development  process. 

Chapter  3  is  composed  of  the  six  case  studies  of  recent 
commerical  and  military  aircraft  which  are  rated  using  the 
proposed  methodology.  The  cases  are  presented  in  the  order  of 
higher  to  lower  systems  engineering  fundamentals  scores,  and  they 
include  the  Boeing  777  commercial  transport,  Lockheed  F-117 
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Stealth  Fighter,  Northrop  B-2  Stealth  Bomber,  McDonnell  Douglas 
C-17  military  transport,  Lear jet  Model  60  business  jet,  and 
McDonnell  Douglas  MD-11  commercial  transport. 

In  Chapter  4,  the  cases  are  briefly  compared  and  the  results  of 
the  ratings  are  summarized,  graphed,  and  analyzed.  The  graphs 
suggest  a  possible  positive  relationship  between  the  systems 
engineering  fundamentals  score  and  the  performance  score  for  each 
case.  The  chapter  also  proposes  possible  refinements  to  the 
methodology  and  suggestions  for  additional  data  generation  and 
analysis. 

Chapter  5  presents  the  results  of  the  validation  run  conducted  by 
nine  engineers  help  identify  weaknesses  in  the  case  studies  and 
evaluation  methodology  so  that  they  could  be  improved.  The 
results  indicate  close  agreement  between  the  group's  average 
scores  and  the  author's  scores,  indicating  reasonable  clarity  of 
the  cases  and  methodology. 
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LEARJST  MODEL  60  DEVELOPMENT  QUESTIONNAIRE  FOR 
CAPT  JAX  MOODY'S  SYSTEMS  ENGINEERING  RESEARCH  PROJECT 

DEVELOPMENT  PROGRAM  FEATURES 


-  Development  Approach 

—  traditional,  skunkworks,  integrated  product  development 

(concurrent  engr)? 

-  organizational  Structure  (functional-matrix  project  team, 
project  team,  design/build  teams,  interface  teams,  other)? 

MiVTrf-vl  _  --- 


-  Average  Size  of  Contractor  Team  during  Development  (including 

major  subcontractors):  ; 

-  Electronic  communication  links  established  between  prime,  subs, 

vendors ,  customers  f  users?  tia  _ _ _ _ * - 

-  Research  cost:  n 

-  Development  cost!  f 


9o 


Him  tp  ^ 


3  SW-'VOMT  /V^r 


—  Funding  profile  per  fiscal  year: 

TW  >*P  fit***'  lit.  *£  (*£<£**2m  _ £ 

-  %  Overrun  of  development  costs: 


A  6£± 


r  *a»ii 


H^rn/ , 


_ Zly-c 


_c.*ju&IL 


-  Date  of  Program  Start: 


qO 

md  d< 


Cr- _ 


MjjT  &*•> 


-  Advanced  Development  start/end  dates:. 


-  Full-scale  Engineering  Development  (FSED)  (or  EMD)  start/  end 

dates :  aw.  fl  £  -  S'**-*  5 3  — — - - 

-  Development^  Length  (years):  k 


-  Behind  Schedule  original  plan  (months):  aa  c  . 


-  pate  of  1st  Production  Unit  Delivery  to  customer:  3  3.  _  •fP’ 

-  Number  of  production  units  ordered/on  °ption:  **  ~ 

-  Production  delivery  profile  by  year:  FY9fc — 2S^ — FY9jT; — 33^ — EY9b- — _ 

?n.  fyoX:  18.  FY9g:  18.  FY93l  18,  USst- .18 

-  Production  cost  (first  unit^:  —  y  *?;>  — - — , - 

-  Flyaway  Cost/  Unit  production  Sale  Prj.ce:  $b.6m  without _pption.e_ 

§pd  escallatioii 


MODELING  AND  ANALYSIS  TOOLS 


-  Simulation  modeling  -  what  for. 

- —  Aerodynamics?  (yes/no) 

— —  What  programs?  How  used?  Used  TRANAIR  CFlj — SOf tvare — — — * — 
proqram  ffull  potential  analysis  cods)  <?n  NASA-Ames  Cray 
♦•r,  model  details  and  calculate  shock  waves  for  transonic 


JUN  30  *94  16:55 


PAGE.  004 
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—  Electrical?  (yes/nol 

- —  What  programs?  How  used?  ct 


^  O1^ 


0  t  A<T*.iv  i 


Hydraulics?  (ves/no) 

-  What  programs?  How  used? 


—  Other? 


How  used? 


Wind  Tunnel  tests  performed?  What  purposes?  "  ” 

HvC.1  <- (>*-*£  rcTTT  Ta  £  f >■)  cal 

0<|  i/a»  1 1  c  U*w  6 cO  •* —  fectijcpO  P>Cift^ _ 


-  Mockups  used?  What  kind/what  for? 

pjKwvi  vP  Moe,t  ■-  ^  /Ho.-'  C^diO  /l-  J~  <  b*j 


-  Breadboards /brassboards  used?  (yes/no)  How  extensive? 


Prototypes  used?  (yes /no) 

—  Advanced  Technology,  FSF.n,  or  Production  prototypes?  One 

engineering  prototype  and  4  preproduction  prototypes - 

were  used  tor  flight  testing. _ The  4  preproaueLion — 


prototypes  were  delivered  to  customers  after  completion  Of 
flight  testing  and  FAA  certification. 

How  close  were  prototype  configurations  to  production  units 
configuration?  (gjLcjh,  Medium,  Low) 

Prototype  delivery  dates  (1st,  2nd,  3rd,  4th,  5th,  6th,  ?th) 


HARDWARE / SOFTWARE  TESTING 

-  Hardware-in-the-loop  (HITL)  simulation  used?  (yes/noj 

-  What  major  components  tested  in  HITL? 


JUN  30  ’94  16=55 
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-  Component/subsystem  testing  -  done  early  in  dev?  (yes/no) 


Use  of  airborne  testbed  for  avionics?  (yes/no) 
—  What  aircraft  and  how  used? 


-  Use  of  ground  test  articles?  (J£fi£/no) 

-  Ground  Testing  (type,  start,  end) 

*  fill  _ 


-  Number  of  Flight  Test  Aircraft:  4 

-  First  Flight  (date):  July  1991 

-  Flight  Testing  -  development  and  certification/operational- 
( type ,  start ,  end ) 

1st  unit: _  M - ,  , 


2nd  unit: 


3rd  unit; 

4th  unit;  function  and  reliability  flight  testing 


-  Flight  Testing  -  total  length  (months):  IS. 

-  Reliability  Testing?  Component  level?  System  level?  Reliability 

flight  testing  (prototype  tAl - 1 - 


-  T,ogietics  Testing?  What  Rind?  Vb  " 


--  Performed  early  or  late  in  developoment? 

—  Major  design  changes  due  to  FSED  (EMU)  testing? _ »^,a 


JUN  30  ’94  16:56 


PAGE. 006 


P07 


SYSTEMS  ENGINEERING  TOOLS f  TECHNIQUES,  AND  REQUIREMENTS 

-  Company  had/has  a  separate  systems  engineering  organization 
workinq  on  the  development  program?  (yes/no) 

-  Design  approach:  Requirements  f lowdown  (top  down)  or  bottoms 

up?  ti«n+ _ ■  _ 

-  Use  of  object  oriented  programming?  (yes/no) 

-  Use  of  Integration  Design/Manufacturing  Technologies  (design 
languages)?  (yes/no)  Prime  contractor?  or  subcontractor  level? 
For  what? 

—  CADD,  CAD,  CAE? _ 

—  Computer  aided  software  engineering  (CASE)?  ucTr  _ 

—  3-D  Modeling?  (yes/no)  What  %  of  design  covered? 

—  3-D  Solid  modeling?  (yes /p&)  What  %  of  design  covered? _ 


CAM,  CIM?  (ves/no) 

CAD/CAM  inteqrated?  (yes/no) 

_ f»«-n/wc _ 

Paperless  Design?  (yes/no)  How 

extensive? 

Paperless  Manufacturing  Floor? 

(ves/no)  How  extensive? 

—  Other? _ _ _ _ 

-  Work  Breakdown  Structure  generated  at  beginning  of  program? 
(yes/noj 

-  Formal  Design  Reviews  planned/conducted?  (yes/no)  With 
customers?  internal?  What  kind? 

—  Systems  requirements  review?  ( yes/no ) 

—  Preliminary  design  review? (ves/no) 

—  Critical  design  review  f system)?  (yes/no) 

—  critical  design  reviews  ( subsystems ) ?  (ves/np) 

—  Software  walk-throuahs?  (yes/no)  ^  C^vo»TCjr 


-  Formalized  configuration  management  system  (yes/jja) 

—  Prime  contractor  has  separate  CM  organization?  ( yes/no ) 

~  Established  baselines/config  items  identified?  (yes/no) 

—  Change  control  system  with  a  change  control  board?  (yes/no) 

—  Formal  requirements  auditing  performed?  (yes/no) 

—  Maintains  status  accounting  of  Engineering  change  requests 
(ECRs),  Failure  Reports  (FRs),  and  Action  Items?  (yes/no) 

— -  Computer  based?  (yes^fi^) 

- —  Failure  Report  and  Corrective  Action  System  (FRACAS) 
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established?  (yes/no) 

—  Level  of  Engineering  Drawings  Req'd/Generated  (3,  4_,  5,  67) 

-  Miscellaneous 

—  Performed  formalized  tradeoff  of  major  doeign  options? 

(yes/aoj 

—  Scheduling  methods/programs  used: 

-  Earned  value  system?  (yes /no) 

— - -  PERT,  Line  of  Jialance,  Gantt ,  Other?  _ 

--  Computerized  cost  accounting  system?  (yes/no) 

—  use  of  statistical  process  control?  (yes/no) 

DOCUMENTATION  REQUIRED/GENERATED 


Did/does  the  program  have: 

-  Systems  Engineering  Management  Plan  or  equivalent?  (yes/jio) 

-  Test  and  Evaluation  Master  Plan  or  equivalent?  (yes/no) 

-  Formal  test  plans  and  procedures?  (yes/no) 

-  Contiguation  Mgt  Plan  or  equivalent?  (yes/no) 

-  Quality  Plan?  (yes/no) 

•  Formal  ILS  plans  or  equivalent?  (yea/no) 

-  Requirements  Documents  and  Specifications 

—  functional  Requirements  Document  (FRD)  or  equivalent? 

(yes/no) 

~  SystenTSpecification  Document  (SSD)  or  equivalent?  (yes/no) 

—  Software  Requirements  Document  (SRD)  or  equivalent?  (yes/no) 

—  Software  Design  Document  (SDD)  or  equivalent?  (yes/no) 

—  Subsystem  specifications?  (yes/no) 

—  Software  unit  folders?  (yes/no) 


DESIGN  HERITAGE,  DESIGN/TECHNOLOGY  DIFFICULTY  MEASURE 

Evaluation  categories:  Off  the  Shelf,  Minor  Modification, 

Redesign,  Major  Redesign,  Original  Design,  Required  Technology 

Breakthrough 

-  overall  design  heritage:  The  yodel  6p  is  a  derivative  of  the 
Model  55  which  was  introduced  in  1981. — ?he  Model  55  underwent 

several  performance  upgrades  (1983  and  1984),  and  then  tfcfi - 

Model  55B  was  introduced  in  1985. - Later  r  -the  M°del  55C  was - 

introduced  which  incorporated  the  earlier  aerodynamic  and - 

performance  improvements — The  Model  60  retains  the - — 

fjE^gm^Ktl  made  to  the  Modal  55.  But  adds  new  elements  to_ths. 

earns  basic  airframe.  Redesign^ 

'  flaps,  spoilers,  ailerons  (Design,  M£r,  Support): 

Based  on  Model  55.  improved  aerodynamics  -  Redesign. 

_  Fuselage  and  nose  cone/radome  (Design,  Mfr,  Support): 
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Fuselage  is  a  3.5  ft  stretched  version  of  the  Model  55 
fuselage.  Redesign . 

—  Tail,  rudder,  horizontal  stabilizers,  elevators  (Design, 

Mfr,  Support):  Essentially  the  same  as  the  Model  55. 

some  minor  aerodynamic  changes.  Minor  modification. 

—  Doors  (Design,  Mfr,  Support):  Essentially  the  same  as  the 
Model  55.  Off-the-shelf. 

—  Landing  Gears,  tires,  and  brakes  (Design,  Mfr,  Support): 
Essentially  the  same  as  the  Model  35.  off-the-shelf. 

—  Fuel  Tanks  (Design,  Mfr,  support): _ 

—  Antennas  (Design,  Mfr,  Support):  _ 

—  Canopy /Windows  (Design.  Mfr.  Support):  Essentially  the  same 

as  the  Model.. Si* _ Off-the-shelf. 

-  Propulsion  — —  — ~  -  —— 

—  Propulsion  Unit  (Design,  Mfr,  Support):  Pratt  &  Whitney _ 

Canada  PW303A  turbofan  engines.  ( I  don't  know  if  it  was _ 

developed  for  the  Model  60  or  It  already  existed1).  Engines 
had  been  flown  before  on  the  P&W  Canada  Boeing  720  testbed  ....... 

aircraft.  _ 

—  Attachment  (Design,  Mfr,  Support): _ ZZZZI^^^ZZZZIZ^^ 

—  Engine  Reveraers  (Design.  Mfr.  Support) :  Rohr  engine  __ 

reversers.  (I  don't  know  anything  of  their  heritage}. 

-  APU  (Design,  Mfr,  support):  None 

-  Avionics  (hardware  and  software) 

—  Cockpit  displays  (Design,  Mix,  Support):  Collins  Fro  Line  4 

avionics  system.  IDS-850  integrated  Display  System. _ ri  don't 

know  if  it  is  the  same  as  the  Model  55.  is  off-the-shelf,  or 
was  developed  specifically  for  the  Model  60. 


—  Navigation/Guidance  units  (Design,  Mfr,  Support): 

—  Communications  (Design,  Mfr,  Support) t+sj 


—  Parfarc  (negign,  Mfr,  Support):  Standard  Collins  WXR-840 - 

Wither  radar,  off-the-shelf. 

—  Flight  safety/collision  avoidance  (Design,  Mfr,  Support): 


—  Automatic  landing  system  (Design,  Mfr,  Support ) 


—  Flight  Computers  and  autopilot (Design,  Mfr^ Support ) : 
Equipped  with  a  single  Collins  FMS-850  flight  management 
system  computer.  (I  don't  know  if  it  is. the  same  as  the — 
Model  55). _ _ _ _ _ 


—  Entertainment  System  (Design,  Mfr,  Support ) ! 


-  IFF  (Design,  Mfr,  Support):  N/A 
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Fire  Control  (Design,  Mfr,  Support):  N/A 
Targeting  System  (Design,  Mfr,  Support):  N/A 
Penetration  Aids  (Design,  Mfr,  Support):  K/A 
Radar  warning  receiver  (Design,  Mfr,  Support):  fi/A 
Avionics  Integration  (Design,  implementation): _ 


—  Others? 


-  Flight.  controls  (Design.  Mfr.  Support): 


-  Pnol  cyctowi  ^esian,  Mir.  Support!:  a5  ?S  «T*wrfr~ 

_ <-/%<-<;  ffl-. - — - - — 

-  Passenger  escape  system  (Design,  Mfr,  Support):  N/A 

-  Environmental  control  system  (heat  S  A/C)  (Design,  Mfr, 
Support):  Essentially  the  same  as  the  Model  55.  Off~the-sh.el.f-i. 

-  oxygen  system  (Design,  Mfr,  support):  Essentially  the  same  as 
the  Model  55.  Off-the-shelf^ 

-  Ejection  seat  (Design,  Mfr,  Support):  N/A 

-  BITE/Central  Integrated  checkout  (Design,  Mfr,  Support): _ 


—  Level  of  BITE  on  aircraft  (High,  Mfidin®lt  Low) 

-  Cargo  Loading/unloading  System  (Design,  Mfr,  Support):  N/A 

-  Aerial  Refueling  System  (Design,  Mfr,  Support) ;  N/A 

-  Weapons  Delivery  Equipment  (Design,  Mfr,  Support):  R/A 


-  Support  Elements  , 

— ~  dew  ground  support  cquipni©n.t/tcot  equipni©nt-  ( Design / 

>ort) ;  Af  r  f  l-.**'*' - 


Mfr, 


Support ) 


New  maintenance  procedures 


(Computer  based? ) :  0  lL 


—  New  training  eguipment/simulators  (Design,  Mfr, Support ) : — 

d - — - 

—  New  training  procedures?  (yes/jjo)  Computer  based?  (yes/noT 

~  n«w  ground  facilities  (Design,  Mfr,  Support): - - 


-  integration  (Design,  Implementation): _ 


L.<nr 


Overall  composite  content  %  by  weight:^ 

Total  number  of  parts  on  aircraft 

—  Not  including  rivets  and  fasteners: _ * — 

—  including  rivets  and  fasteners :  \ _  . . „„ 

Design  remained  stable  after  start  of  production?  (Production 

configuration  consistency)  Y<*  - 

—  How  stable?  ( flight- Medium ,  Low)  ,  .  j  later 

__  aj-o  there  major  difference  between  the  earlier  and  later 

production  deliverables  due  to  quality  -  rework  and  repair 

issues?  )  rt  bL ae. — ^fULS^JL - f - - - — - - 


eXS 
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—  Due  to  redesigns  to  correct  performance  requirements 
shortfalls?  c _ _  _ _ 


MISSIOK  DIFFICULTY  MEASURE  (Requirements  or  actuals) 

-  Range  (unrefueled):  At Wfi  NM 

-  payload  (lbs)  (p^gengers,  cargo,  weapons): 


-  Fuel  Capacity: 


^bs  n<ti©  ft-*"’* 


Std  empty  weight:  13,750  lbs 
Maximum  takeoff  weight:  22.750  lbs  c 
Payload  fraction  (payload/max  t/o  wt): 
Top  speed:  yi-»oc* 

Cruise  speed:  Mach  . 8 1 

Mission  reliability/avai lability : ___ 

Maneuverability  (deg  roll/sec); _ 

Runway  (unpaved  landing/takeoff):  No 
Cargo  airdrop:  No 
Aerial  refuelings  No 
Reduced  radar  cross  section:  No 
Weapon  delivery:  No 


»7cW<  t-Ai(c9  jo 
•  »  «/d*9o  m  CLj,'/,  tJ  £ 


REQUIREMENTS  compliance  measure 

-  Areas  of  major  non-compliance/reqmts  reductions:  (For  jffijMRplgi 
did  the  Model  60  meet  range,  engine  efficiency,  and  weight - 

requirements  /goals?_l  - - - - 


DEVELOPMENT  ENVIRONMENT  MEASURE 

-  Extent  of  customer/market  involvment  in  defining  requirement 
for  the  new  aircraft?  - f a 

_  Requirements  remained  stable  throughout  deveiopmont?  (^es/no) ~ 
—  List  major  changes?  - - — . — — 


I  Spin?  fording  with 

-  I’TntlV  «  eKST 

firm  orders  for  the  breakeven  quantity  were  received. 

-  organizational  structure  and  personnel  stability.  (High, 

Medium,  Low) 
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-  Stability  of  subcontractor/vendor  team?  (Ijiflh,  Medium,  Low) 

-  Company  background  in  developing  the  aircrafts  Lear jet  has  been 

designing  and  building  business  lets  for  ovsr  39  ve.ars._The - 

Model  60  is  a  derivative  of  the  Model  55 

-  Government  oversight  of  development?  (High,  Medium,  Law ,  none ) 

-  Government  oversight  of  flight  testing/certification?  (high, 
Medium,  JLa*,  none) 

-  i^vel  of  customer/user  interface  with  developers  (High,  Medium, 

Low) 

-  Level  of  security  measures  (High,  Medium,  Log) 

-  Initial  tooling  established  for  limited  or  long  term 

production?  qjj  !±°  ugh* - - 
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